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1.4 


Introduction 


A.  A.  Markov  (1856-1922)  was  a  prominent  member  of  a  group  of  Russian 
mathematicians  at  the  University  of  St  Petersburg,  who,  during  the  second  half  of 
the  nineteenth  century,  contributed  extensively  to  probability  theory.  Markov’s 
principal  works  were  concerned  with  limit  theorems  for  sums  of  random  variables, 
both  independent  and  dependent;  he  initiated  the  study  of  dependent  random 
variables.  Between  1907  and  1912,  he  studied  the  applicability  of  limit  theorems, 
such  as  the  Central  Limit  Theorem,  to  a  sequence  of  random  variables  forming  a 
chain .  Such  a  ‘chain’  is  now  referred  to  as  a  Markov  chain,  and  has  become  an 
important  tool  in  probability  theory.  Markov  chains  are  used  to  model  a  wide 
range  of  phenomena  in  many  branches  of  science,  engineering,  economics  and  the 
social  sciences;  several  examples  will  be  discussed  in  Section  1  of  this  unit. 

In  1913  Markov  published  a  paper  called  ‘An  example  of  statistical  investigation  of 
the  poem  “Eugene  Onegin’”.  In  this  paper  he  used  a  Markov  chain  model  to 
study  the  sequence  of  vowels  and  consonants  in  Russian  words;  he  took  a 
sequence  of  20000  letters  from  Pushkin’s  ‘Eugene  Onegin’  and  showed  that  this 
may  be  modelled  by  a  simple  Markov  chain  in  which  the  probability  that  a 
randomly-chosen  letter  is  a  vowel  depends  on  whether  or  not  the  previous  letter  is 
a  vowel.  In  recent  years,  Markov  chain  models  have  been  used  to  carry  out  similar 
investigations  in  other  languages.  Markov’s  model  and  his  results  will  be  discussed 
in  more  detail  in  Section  2. 

This  unit  is  the  last  of  three  units  on  stochastic  processes  with  discrete  state  space 
and  in  discrete  time.  In  the  first  section  of  Unit  5  a  random  walk  was  defined  and 
those  of  its  features  that  are  possessed  by  the  class  of  stochastic  processes  known 
as  Markov  chains  were  noted;  in  fact,  the  branching  processes  of  Unit  4  and  the 
random  walks  of  Unit  5  are  all  examples  of  Markov  chains.  Some  of  the  ideas 
discussed  in  this  unit  will  be  familiar  to  you  from  Unit  5;  for  example,  the  ideas  of 
a  limiting  distribution  (Section  2)  and  of  the  recurrence  or  transience  of  states 
(Sections  4  and  5)  will  be  extended  to  Markov  chains;  and,  once  again,  generating 
functions  will  be  used  to  obtain  a  criterion  for  recurrence  (Section  5).  Many 
properties  of  Markov  chains  will  be  illustrated  by  examples  of  random  walks  from 
Unit  5,  so  you  will  find  it  helpful  to  have  that  unit  handy  as  you  work  through 
some  sections  of  this  unit;  you  will  find  it  particularly  useful  for  Subsections  1.1 
and  5.1. 

After  some  notation  and  a  few  basic  ideas  have  been  introduced  in  Section  1, 
together  with  various  applications  of  Markov  chains,  the  long-run  behaviour  of  a 
Markov  chain  will  be  investigated  in  Section  2.  What  proportion  of  time  does  a 
Markov  chain  spend  in  each  state?  Does  the  distribution  of  the  state  occupied 
‘settle  down’  in  some  way  when  the  chain  has  been  running  for  a  long  time?  These 
are  two  of  the  questions  that  we  shall  consider.  The  ‘equilibrium’  of  a  Markov 
chain  will  be  discussed  and,  in  particular,  the  idea  of  a  limiting  distribution;  this 
will  be  considered  in  more  detail  than  it  was  for  the  gambling  scenario  in  Unit  5; 
and  a  general  method  will  be  obtained  for  finding  a  limiting  distribution  when  it 
exists.  But  not  until  Subsection  3.2,  after  various  properties  of  Markov  chains  have 
been  discussed,  will  it  be  possible  to  give  conditions  for  a  Markov  chain  to  have  a 
limiting  distribution.  At  the  end  of  Section  4  there  is  an  audio-tape  session  in 
which  you  are  asked  to  work  through  some  examples  which  illustrate  and  bring 
together  the  various  ideas  and  results  of  Sections  3  and  4.  In  Section  5  the  ideas  of 
transience  and  recurrence  are  discussed  in  more  detail  than  previously.  There  is  no 
video-cassette  component  associated  with  this  unit. 

Most  of  the  ideas  and  results  of  this  unit  are  contained  in  the  first  three  sections. 
The  first  two  sections  are  longer  than  average;  however,  Section  4  is  very  short, 
consisting  principally  of  the  audio-tape  session,  and  Section  5  contains  very  few 
new  ideas.  You  will  find  some  of  the  material  in  Subsection  5.1  very 
straightforward  if  you  studied  Section  4  of  Unit  5  carefully;  and  if  you  did  not,  or 
if  you  found  that  section  of  Unit  5  difficult,  then  working  through  Subsection  5.1 
should  help  you  to  sort  out  some  of  the  ideas  you  met  there. 


Before  you  study  this  unit,  you  should  be  familiar  with  matrix  notation  and  be 
able  to  multiply  matrices;  all  the  information  you  require  is  in  the  Handbook.  You 
will  also  need  to  be  able  to  find  a  probability  generating  function  (p.g.f.)  and  use  it 
to  find  the  mean  of  a  distribution;  p.g.f.’s  were  revised  in  Unit  4,  and  a  summary  of 
their  properties  is  given  in  the  Handbook. 

The  study  of  Markov  chains  is  an  important  branch  of  probability  theory  with  a 
wide  literature;  so,  in  a  single  unit,  it  is  possible  to  introduce  only  a  few  ideas  and 
to  discuss  a  few  of  the  problems  associated  with  Markov  chains.  Furthermore, 
many  apparently  simple  results  require  quite  sophisticated  mathematical 
arguments  to  prove  them;  the  approach  adopted  here  to  such  results  is  to  show 
that  they  are  reasonable  but  to  omit  the  proofs.  (The  bibliography  in  the  Course 
Guide  includes  textbooks  which  cover  these  proofs.) 


1  Markov  chains  and  their  applications 


You  have  already  met  examples  of  Markov  chains  in  the  branching  processes  of 
Unit  4  and  the  random  walks  of  Unit  5.  Markov  chains  were  introduced  in 
Subsection  1.3  of  Unit  5;  the  ideas  discussed  there  are  revised  briefly  at  the 
beginning  of  this  section  and  a  definition  of  a  Markov  chain  is  given.  Then  the 
transition  matrix  of  a  Markov  chain  is  introduced;  this  is  simply  a  convenient  way 
of  writing  down  all  the  transition  probabilities.  These  ideas  are  illustrated  for 
several  examples  from  Unit  5,  so  you  may  find  it  helpful  to  have  that  unit  handy 
as  you  work  through  Subsection  1.1.  In  Subsection  1.2  two  applications  of  Markov 
chains  are  described — in  modelling  weather  patterns  and  for  studying  job 
mobility;  these  examples  are  used  to  introduce  some  ideas  and  results  concerning 
transitions  involving  n  steps. 


1.1  What  is  a  Markov  chain? 


Before  considering  a  definition  for  a  Markov  chain,  we  shall  look  briefly  at  two 
examples  from  Unit  5:  the  gambler’s  ruin  and  the  animal-collecting  problem. 

Example  1.1  The  gambler’s  ruin 

In  the  gambler’s  ruin,  Gary  has  £k  initially  (0  <  k  <  a)  and  his  opponent  has 
£(a  -  k),  so  they  have  £a  between  them.  In  each  game  Gary  wins  £1  with 
probability  p  or  loses  £1  with  probability  q;  Xn  is  Gary’s  fortune  after  n  games, 
and  the  state  space  of  the  process  {Xn;  n  =  0, 1,  ...}  is  (0, 1,  ...,  a}.  The  transition 
probabilities  are  shown  in  Figure  1.1.  □ 


Figure  1.1  Transition  probabilities  in  the  gambler’s  ruin 


Example  1.2  Collecting  animals 

In  a  breakfast  cereal  promotion,  a  plastic  animal  is  given  away  with  each  packet. 
There  are  three  different  animals,  and  each  packet  is  as  likely  to  contain  any  one 
of  them  as  any  other;  X„  represents  the  number  of  different  animals  that  a 
particular  family  has  obtained  after  buying  n  packets  of  cereal.  The  transition 
probabilities  are  shown  in  Figure  1.2.  □ 
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Figure  1.2  Transition  probabilities 
for  collecting  animals 


Unit  5,  Section  2 


p  +  q=l 


Unit  5,  Example  1.7 


The  processes  m  these  two  examples  have  two  properties  in  common:  they  possess 
the  Markov  property  and  they  have  stationary  transition  probabilities;  both 
properties  were  considered  in  Unit  5,  Subsection  1.3. 

A  stochastic  process  has  the  Markov  property  if  the  probability  of  any  future 
behaviour  of  the  process,  when  its  current  state  is  known,  is  not  altered  by  any 
additional  information  about  its  past  behaviour.  The  processes  in  Examples  1.1 
and  1.2  have  this  property.  For  example,  in  the  gambler’s  ruin,  if  Gary  has  £i  after 
n  games,  then  the  probability  that  he  has  £j  after  the  next  game  is  not  affected  by 
his  fortune  at  any  earlier  stage.  And  in  the  animal-collecting  problem,  if  x  different 
animals  have  been  collected  from  the  first  n  packets,  then  the  probability  that 
another  different  animal  is  obtained  in  the  next  packet  is  not  altered  by  any  extra 
information  about  the  number  collected  at  earlier  stages. 

Also,  in  both  examples,  for  any  given  states  i  and  j,  the  transition  probability 
^(^+i  =j\Xn  =  i )  has  the  same  value  for  all  n ;  that  is,  it  is  constant  over  time. 

So  the  processes  have  stationary  transition  probabilities. 

In  fact,  all  the  processes  in  this  unit  have  both  the  Markov  property  and 
stationary  transition  probabilities;  these  two  properties  are  stated  formally  in  the 
following  definition. 


A  stochastic  process  {X„;  n  =  0, 1,  ...}  in  discrete  time  with  a  finite  or  infinite 
discrete  state  space  S  is  a  Markov  chain  with  stationary  transition  probabilities 
if  it  satisfies  the  following  two  conditions. 

(1) 

For  each  n  >  1,  if  A  is  an  event  depending  only  on  any  subset  of 
{ Xn  —  i ’ X„  —  2 ,  •••,  X0},  then,  for  any  states  i  and  j  in  S, 

p(Xn+1  =j\Xn  =  i  and  A)  =  P(Xn+1  =j\X„  =  i ). 

(U) 

(2) 

For  any  given  states  i  and  j, 

p(Xn+1  =j  \Xn  =  i)  has  the  same  value  for  all  n  >  0. 

(1.2) 

Condition  (1)  is  the  Markov  property,  of  which  the  following  is  a  more  general 
formulation  that  is  also  useful  in  practice. 


(1A)  For  each  n  >  1  and  m  >  1,  if  A  is  an  event  depending  only  on  any 
subset  of  {Xn_1,X„-2,  . ..,  A'o},  then,  for  any  states  i  and  j  in  S, 

P(Xn+m  =j\X„  =  i  and  A)  =  P(Xn+m  =j\Xn  =  i).  (1.3) 


The  event  A  in  Equations  (1.1)  and  (1.3)  may  include  information  on  the  state  of 
the  process  at  time  n  -  1  only,  or  at  the  start  only;  or  it  may  include  the  whole 
history  of  the  process  prior  to  time  n,  or  any  part  of  this  history.  Whatever  the 
case,  this  information  does  not  affect  the  probability  of  any  future  behaviour  if  the 
state  of  the  process  at  time  n  is  known.  So  given  any  information  about  the  past 
behaviour  of  the  process,  only  the  most  recent  information  is  needed  to  determine 
the  probability  of  any  future  behaviour.  For  example, 

p(X s  =j\X4  =  i  and  X3  =  a)  =  P(X5  =j\X4  =  i ), 

p(*4  =j\X2  =  i  and  X0  =  k)  =  P(X4  =  j\X2  =  i ) 
and  P(X10  =j\X5  =  i,X4  =  a,...,  X0  =  k)  =  P(X10  =  j \X5  =  i). 

The  Markov  property  is  used  frequently  in  this  unit,  both  in  the  general  form  (1A) 
and  in  the  form  (1). 


Condition  (2)  requires  that  the  transition  probabilities  P(Xn+1  =j  \X„  =  i)  are 
stationary;  that  is,  for  any  given  states  i  and  7,  the  probability  P(Xn  +  1  =j\Xn  =  i) 
is  independent  of  n  and  hence  this  (one-step)  transition  probability  can  be  denoted 
by  in  fact,  on  many  occasions  the  comma  may  be  omitted  without  introducing 
ambiguity,  thus 

Pij  =  P(Xn  + 1  =j\Xn  =  i). 

Since  discussion  in  this  unit  is  limited  to  processes  whose  transition  probabilities 
are  stationary,  we  shall  follow  the  common  practice  and,  for  simplicity,  call  a 
process  satisfying  Conditions  (1)  and  (2)  a  Markov  chain;  and,  whenever  we  use  the 
term  Markov  chain,  we  shall  mean  a  Markov  chain  with  stationary  transition 
probabilities. 

The  two  conditions  in  the  definition  of  a  Markov  chain  make  the  calculation  of 
many  probabilities  associated  with  such  a  process  straightforward.  For  example,  if 
the  process  is  in  state  i  at  some  time  n,  then  the  probability  that  the  process  moves 
from  state  i  to  state  j  and  then  from  j  to  k  in  consecutive  steps  is 

+  i=7  and  Xn + 2  =  k  |  X„  =  i) 

=  P(Xn  +  2  =  k\Xn+1  =j  and  Xn  =  i)P(Xn  +  1  =j\Xn  =  i) 

=  P(X„  +  2  —  k  |  Xn+ 1  =  j)  P(Xn+1  =  j  |  X„  =  i), 
using  the  Markov  property  (1).  Then,  from  Condition  (2),  we  have 
P(Xn  +  l  =7  and  Xn  +  2  =  k\Xn  =  i) 

=  P(Xn+1=k\Xn=j)P(Xn+1  =j\Xn  =  i) 

=  PjkPu  =  PijPjk •  (1.4) 

That  is,  the  probability  of  going  from  i  to  j  to  k  is  equal  to  the  product  of  the 
probabilities  of  moving  from  i  to  j  and  from  j  to  k.  More  generally,  the  probability 
that,  starting  from  state  i,  the  process  moves  consecutively  through  the  states 
7,/c,  ...,  y,z,  is  pupjk  ...  pyg,  as  you  are  asked  to  show  in  Question  1.1. 

Question  1.1  Use  Conditions  (1)  and  (2)  to  show  that 

p(X i  =j,X2  =  k,...,X„-1=y,Xn  =  z\X0  =  i)  =  PijPjk  . . .  pyz, 
by  applying  the  rule  P(A  n  B)  =  P(A  \  B)  P(B),  all  events  being  conditioned  on 
*o  =  i-  □ 


Transition  matrices 


Clearly,  the  transition  probabilities  pu  are  fundamental  in  describing  how  a 
Markov  chain  behaves;  it  is  customary  and  often  convenient  to  arrange  these 
numbers  in  a  matrix  P  whose  (i,  j)  entry  is  py.  For  example,  if  the  state  space  S  is 
the  finite  set  (0, 1,  ...,  a},  then  P  is  the  {a  +  1)  x  (a  +  1)  matrix 


Poo 

Pot 

P02 

•  POa 

P 10 

Pit 

Pl2 

•  Pla 

P20 

P21 

P22  • 

■  P2a 

_PaO 

Pal 

Pa2  • 

Paa  _ 

If  S  is  the  infinite  set  (0, 1,2, ...},  then  P 


Poo 

P  01 

P  02 

P 10 

Pll 

P 12 

P20 

P21 

P22 

is  the  infinite  matrix 


The  matrix  P  is  called  the  transition  matrix  of  the  Markov  chain. 


The  comma  is  needed  in  p12,i,  for 
example. 


The  rule  P(A  n  B)  =  P(A  |  B )  P(B ) 
has  been  used  with  all  events 
conditioned  on  Xn  =  i;  so, 
P(AnB\Xn  =  i) 

=  P(A  \  B  n  Xn  =  i)  P(B\Xn  =  i). 


Calculations  involving  matrices  of 
infinite  order  are  carried  out  using 
the  same  rules  as  for  matrices  of 
finite  order. 
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Example  1.1,  continued 


The  transition  probabilities  in  the  gambler’s  ruin  are  shown  in  Figure  1.1,  which  is 
repeated  below. 


Figure  1.1  Transition  probabilities  in  the  gambler’s  ruin 


For  0  <  i  <  a. 


Pu+i  =  P  and  Pi,i  —  i  =  q; 

also,  since  the  match  stops  when  one  or  other  player  accumulates  £a, 

Poo  -  1  and  paa=  1 ; 

all  other  transition  probabilities  are  zero.  This  process  is  a  Markov  chain  and  its 
transition  matrix  is 


0  12  3  4 

0  10  0  0  0 

/  q  0  p  0  0 

2  0  q  0  p  0 


0 

0 

0 


P  = 


□ 


a  —  1 


a 


0 

0 


0  q  0  p 
0  0  0  1 


Example  1.2,  continued 


Figure  1.2  Transition  probabilities  for 
collecting  animals 


Sometimes,  as  here,  state  labels  are 
included  for  the  rows  and  columns; 
this  is  done  to  make  it  easy  to 
identify  individual  transition 
probabilities.  For  example,  pl2  =  p 
is  the  transition  probability  from 
state  1  to  state  2. 


The  transition  probabilities  for  the  animal-collecting  problem  are  shown  in 
Figure  1.2  (repeated  above).  The  process  is  a  Markov  chain,  and  its  transition 


matrix  is 

0 

1 

2  3 

0 

0 

1 

0  0 

p  1 

0 

i 

!  0 

P  =  2 

0 

0 

t  3 

■  □ 

3 

0 

0 

0  1 
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Example  1.3  A  Markov  chain  model  for  success  runs  in  Bernoulli  trials 

In  a  Bernoulli  process  in  which  each  trial  results  in  a  success  with  probability  p,  let 
Xn  denote  the  length  (so  far)  of  the  current  run  of  successes  after  n  trials;  that  is, 

Xn  is  the  number  of  trials  since  the  last  failure  (or  since  the  start  if  there  has  been 
no  failure).  The  state  space  S  for  the  process  {X„;  n  =  0, 1,  . . .}  is  the  set  of  non¬ 
negative  integers,  S’  =  {0, 1,2,  If,  at  time  n,  the  process  is  in  state  i,  then  at 
time  n+  1,  after  the  next  trial,  it  will  be  in  state  i  +  1  if  that  trial  results  in  a 
success— this  has  probability  p— or  it  will  return  to  state  0  if  the  outcome  of  the 
trial  is  a  failure  this  has  probability  q  =  1  —  p;  no  other  transition  is  possible. 

So,  for  n  >  0  and  i  >  0, 

Pi,i+  1  —  P{X„+  !  =  i  -f-  1 1  X„  =  i)  =  p, 

Pio  =  P(Xn+ 1  =  0 1  X„  =  i)  =  q, 
and  for  j  ^  0,  i  +  1, 

Pij  =  P(Xn+l  —  j |  Xn  =  i )  =  0. 

The  Markov  chain  {Xn;  n  =  0, 1,  ...}  is  illustrated  in  Figure  1.3. 


The  transition  matrix 

is 

<7 

P 

0 

0 

0 

0  ..." 

<7 

0 

P 

0 

0 

0  ... 

P  = 

<7 

0 

0 

P 

0 

0  ... 

•  □ 

<7 

0 

0 

0 

P 

0  ... 

The  entries  in  a  transition  matrix  have  two  important  properties.  Firstly,  since  ptJ 
is  a  probability  it  cannot  be  negative,  so  all  the  entries  in  a  transition  matrix  are 
non-negative;  that  is, 

Pq  >  0  for  all  i  and  j  in  S.  (1-5)  Also,  of  course,  ptj  <  1. 

Secondly,  from  being  in  state  i  at  some  time  n,  at  the  next  stage  the  Markov  chain 
must  either  remain  in  state  i  or  move  to  one  of  the  other  states  in  S;  so  the  sum  of 
the  probabilities  of  the  possible  transitions  from  i  is  equal  to  1.  This  is  true  for 
each  i  in  S,  so 

Yj  Pij  =  1  for  each  i  in  S.  (1.6) 

jeS 

Thus,  the  sum  of  the  entries  in  each  row  of  a  transition  matrix  is  1 ;  using  this 
property  can  sometimes  save  time  when  calculating  transition  probabilities.  For  a 
diagram  representing  a  Markov  chain,  Equation  (1.6)  says  that  the  probabilities  on 
the  arrows  leaving  each  state  must  add  up  to  1.  For  example,  in  Figures  1.1,  1.2 
and  1.3,  the  sum  of  the  probabilities  on  the  arrows  leaving  each  state  is  1. 

Question  1.2  For  each  of  the  following  Markov  chains  {Xn;  n  =  0, 1,  write 
down  the  transition  matrix  P. 

(i)  Gambling  against  a  casino  Suppose  Gary  is  gambling  against  an  opponent 
with  unlimited  resources,  and  in  each  game  he  wins  £1  with  probability  p  or 
loses  £1  with  probability  q.  Let  Xn  be  his  fortune  after  n  games. 

(ii)  Let  {Xn;  n  =  0, 1,  ...}  be  the  simple  random  walk  with  two  reflecting  barriers 

shown  in  Figure  1.4.  □ 


Figure  1.4  A  simple  random  walk  with  two  reflecting  barriers 
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Question  1.3  Suppose  Gary  initially  has  £k  and  he  is  gambling  against  his 
benevolent  uncle  who  starts  with  £(20  -  k),  so  they  have  £20  between  them.  If 
Gary  ever  loses  his  last  pound,  his  uncle  returns  it  to  him  and  they  play  on;  but,  if 
his  uncle  ever  loses  all  his  money,  he  allows  Gary  to  keep  it  and  they  stop  playing. 
Let  Xn  be  Gary’s  fortune  after  n  games.  If  Gary  wins  £1  with  probability  p  or  loses 
£1  with  probability  q  in  each  game  (q  =  1  -  p ),  draw  a  diagram  showing  the 
transition  probabilities  and  write  down  the  transition  matrix  for  the  Markov  chain 
{X„;n  =  0,1,...}.  □ 

Question  1.4  For  each  of  the  following  Markov  chains  { Xn;  n  =  0, 1,  ...},  draw  a 
diagram  showing  the  transition  probabilities  and  write  down  the  transition 
matrix  P. 

(i)  Random  placement  of  balls  In  each  of  a  sequence  of  independent  trials,  a  ball 
is  placed  at  random  in  one  of  six  containers.  Let  Xn  be  the  number  of 
containers  that  have  at  least  one  ball  in  them  after  n  trials. 

(ii)  A  white  rat  is  put  into  the  maze  shown  in  Figure  1.5.  The  rat  moves  through 
the  compartments  at  random;  that  is,  if  there  are  k  ways  to  leave  a 
compartment,  it  chooses  each  with  probability  1  jk.  It  makes  one  change  of 
compartment  at  each  time  point.  Let  X„  be  the  number  of  the  compartment 
that  the  rat  occupies  at  time  n.  □ 

In  each  of  the  Markov  chains  discussed  so  far,  it  has  been  possible  to  write  down 
explicitly  the  form  of  the  transition  matrix.  However,  there  are  Markov  chains  of 
importance  for  which  finding  an  explicit  form  for  the  transition  probabilities,  and 
hence  writing  down  the  transition  matrix,  is  difficult.  A  Galton- Watson  branching 
process  is  of  this  type;  the  distribution  of  the  number  of  individuals  in  any 
particular  generation  depends  only  on  the  size  of  the  previous  generation.  If  Z„  is 
the  size  of  the  nth  generation,  then  the  process  (Z„;  n  =  0, 1,  ...}  is  a  Markov 
chain,  and 

Pu  =  P(Zn  +  i  =  j  |  Z„  =  i)  =  P(X1  +  X2  +  . . .  +  X/  =j), 
where  Xk  is  a  random  variable  representing  the  number  of  offspring  of  the  kth 
individual  in  the  nth  generation.  If  each  Xk  has  p.g.f.  I7(s),  then  Xx  +  X2  +  ...  +  Xt 
has  p.g.f.  (II(s))',  so  Pij  is  the  coefficient  of  s'  in  (n(s))1'.  Although  in  principle  we  can 
obtain  pu,  in  practice  it  is  usually  difficult  to  express  this  probability  explicitly. 

Your  attention  has  been  drawn  to  this  example  to  emphasize  that  not  all  Markov 
chain  theory  involves  matrices,  although  much  of  the  material  in  this  unit  does. 


1.2  Transition  probabilities  and  transition  matrices 

Example  1.4  Melbourne  weather 

In  Melbourne,  during  the  first  three  months  of  1983,  57  dry  days  were  followed  by 
dry  days  and  12  by  wet  days;  8  wet  days  were  followed  by  wet  days  and  12  by  dry 
days.  These  data  are  summarized  in  Table  1.1. 


Table  1.1 


Weather  on 

the  following  day 

Dry 

Wet 

Total 

Weather  on  one  day 

57 

12 

69 

3  Wet 

12 

8 

20 

A  simple  model  for  predicting  whether  any  particular  day  will  be  dry  or  wet  is 
provided  by  a  Bernoulli  process;  the  basic  assumptions  are  that  the  weather  on 
any  given  day  is  independent  of  the  weather  on  any  other  day,  and  there  is  a 
constant  probability  p  that  it  will  rain.  However,  this  model  does  not  provide  an 
adequate  fit  for  the  observed  data  of  Table  1.1;  it  appears  that  a  wet  day  is  more 
likely  to  occur  after  a  wet  day  (8  of  20)  than  it  is  after  a  dry  day  (12  of  69).  A 
Markov  chain  model  allows  for  some  dependence  between  consecutive  days;  the 
probability  that  it  rains  on  any  given  day  depends  on  whether  or  not  it  has  rained 


Figure  1.5  A  maze 
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on  the  preceding  day.  If  we  take  dry  as  state  0  and  wet  as  state  1,  and  estimate  the 
transition  probabilities  using  relative  frequencies  calculated  from  Table  1.1,  we 
obtain  the  following  transition  matrix  P: 

0  1 

0  I- 0.826  0.174] 

1  0.600  0.400  '  Frora  here  on  we  'gnore  the  fact 

•  that  the  entries  in  P  are  estimates. 

Thus  the  probability  p00  that  a  dry  day  will  be  followed  by  another  dry  day  is 
0.826  (to  three  decimal  places)  and  the  probability  pi0  that  a  wet  day  will  be 
followed  by  a  dry  day  is  0.600.  So  this  model  predicts  that  a  dry  day  is  more  likely 
after  a  dry  day  than  after  a  wet  day.  But  what  about  after  two  days,  or  after  a 
week?  Is  it  more  or  less  likely  to  be  dry  in  two  days’  time  if  it  is  dry  today  than  if 
it  is  wet  today?  And  what  is  the  probability  of  rain  in  a  week’s  time?  How  much 
does  today’s  weather  affect  our  predictions  for  a  week  ahead?  To  answer  these 
questions  we  must  first  calculate  the  probabilities  of  a  dry  day  two  days  after  a  dry 
day  and  two  days  after  a  wet  day;  then  we  must  find  the  corresponding 
probabilities  for  seven  days’  time. 

Let  X„  represent  the  state  of  weather  on  day  n:  Xn  =  0  (dry)  or  Xn  =  1  (wet);  then 
the  probability  that  it  will  be  dry  in  two  days’  time  if  it  is  dry  today  (n  =  0)  is 
P(X2  =  0 1 Z0  =  0).  This  probability  is  the  sum  of  the  probability  that  a  dry  day 
today  is  followed  by  two  more  dry  days  and  the  probability  that  a  dry  day  today 
is  followed  by  a  wet  day  and  then  a  dry  day;  that  is, 

P(X2  =  0\xo  =  0)  =  P(X2  =  0  and  X,  =  0 1 AT0  =  0) 

+  P(X2  =  Oand  Xx  =  l|X0  =  0) 

=  PooPoo  +  PoiPio,  using  Result  (1.4)  twice, 

~  0.787. 

Similarly,  -  ' 

P(X2  =  0|  X0  =  1)  =  pl0p00  +  pupl0 
~  0.736. 

So  a  dry  day  is  more  likely  two  days  after  a  dry  day  than  it  is  two  days  after  a  wet 
day,  but  not  by  very  much.  This  method  could  be  extended  to  find  the 
corresponding  probabilities  for  seven  days’  time,  but  the  calculation  would  soon 
become  very  involved  and  it  would  not  be  difficult  to  make  a  mistake  by  failing  to 
consider  every  possible  pattern  of  weather  for  the  intervening  days.  A  simpler  and 
more  reliable  method  for  calculating  such  probabilities  will  be  introduced;  this 
makes  use  of  the  transition  matrix  P  and  involves  higher-order  transition 
probabilities,  i.e.  transitions  involving  more  than  one  step.  □ 

The  w-step  transition  probability  of  a  Markov  chain,  denoted  pfj  or  pff,  is  the 
probability  that  the  process  goes  from  state  i  to  state  j  in  n  transitions;  that  is,  for 
integers  m,  n  >  0, 

PW  =  p(Xm+„  =j\Xm  =  i). 

So,  in  particular,  p\f  =  P(X„  =j  |  X0  =  i).  The  matrix  of  n-step  transition 
probabilities  is  denoted  P(n).  In  particular,  since  pjj*  =  pu,  we  have  P(1)  =  P. 

The  probability  p\^  is  the  sum  of  the  probabilities  of  all  possible  paths  going  from 
state  i  to  state  j  in  n  steps.  For  example,  to  get  from  state  i  to  state  j  in  two  steps, 
the  process  must  visit  some  state  k  at  the  first  step  and  move  from  there  to  state  j; 
such  a  path  has  probability  pikpkj,  so  summing  over  all  possible  states  visited  at 
the  first  step,  we  have 

P\f  =  I  PikPky  I 

keS 

The  right-hand  side  of  this  formula  is  the  ( i ,  j)  element  of  the  matrix  product 
PP  =  P2;  that  is, 

Pi f  =  (P%,  | 

so  the  two-step  transition  probabilities  are  the  entries  in  the  matrix  P2;  that  is, 

p(2)  _  p2 
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More  generally,  conditioning  on  the  state  occupied  after  the  first  step  and  applying 
the  Theorem  of  Total  Probability,  we  have 

Pu  =  Z  P(Xn  =j\X,  =  k  and  Xo  =  i)P{X1  =k\X0  =  i) 

keS 

—  Z  PikP(Xn  =j\X1  =  k),  by  Markov  property  (1). 

keS 

That  is, 

Pu=  Z  PikP(kYl)-  (1.7) 

keS 

This  result  is  a  special  case  of  the  Chapman-Kolmogorov  equations  for  a  Markov 
chain,  which  state  that,  for  i,  jsS  and  any  integers  m,  n  >  0, 


_(m  +  n) 

rij 


I  P'S' pil- 

keS 


(1.8) 


That  is,  the  probability  of  going  from  state  i  to  state  j  in  m  +  n  transitions  is  the 
sum  over  all  possible  states  k  of  the  probabilities  of  going  from  i  to  k  in  m  steps 
and  from  k  to  j  in  n  steps. 

Question  1.5  By  applying  the  Theorem  of  Total  Probability,  conditioning  on  the 
state  occupied  by  the  Markov  chain  after  the  first  m  transitions,  obtain  the 
Chapman-Kolmogorov  equations  (1.8).  □ 

The  Chapman-Kolmogorov  equations  provide  a  convenient  form  for  P(,,),  the 
matrix  of  n-step  transition  probabilities.  The  right-hand  side  of  Equation  (1.8)  is 
the  (ij)  element  of  the  matrix  product  P<m)P(">;  So,  for  i,jeS,  we  have 

(P,m +  «))._  =  (p(m)p(n)^__ 

and  hence,  for  m,  n  >  0, 

p(m  +  n)  _  p(m)p(n) 

In  particular,  the  special  case  (1.7)  gives 

p(»)  _  p(l)p(n-l)  _  pp(n-l) 

Using  this  recursively,  we  obtain 

P<”)  =  pp("  _  1)  _  ppp('i  -  2)  _  _  pp  p  __  pn 

So  the  matrix  of  n-step  transition  probabilities  is  found  by  calculating  the  nth 
power  of  the  transition  matrix  P;  that  is, 


for  n  >  1,  P(n)  =  P”. 


We  make  use  of  this  result  in  the  following  examples. 


Example  1.4  Melbourne  weather,  continued 


The  2-step  transition  probabilities  are  the  entries  in  P2  and,  to  three  decimal 
places, 


p2 


0.826 

0.174 

0.600 

0.400 

"0.787 

0.213 

0.736 

0.264 

Hence,  the  probability  that  it  will  be  dry  in  two  days’  time  is  0.787  if  it  is  dry 
today  and  0.736  if  it  is  wet  today;  these  are  the  values  which  we  calculated  before. 


Sydney  Chapman  (1888-1970), 
English  applied  mathematician; 

A.  N.  Kolmogorov  (1903-1987), 
eminent  Russian  mathematician  and 
one  of  the  founders  of  modern 
probability  theory. 


This  result  is  used  in  M101 
Block  IV. 
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The  corresponding  probabilities  for  a  week’s  time  are  entries  in  P7 
decimal  places, 

C 

C  oi 


p2p2  = 


0.776 

0.774 


0.224 

0.226 


b 

T  ol  Ls 


.  Now,  to  three 


p6  —  P4P“ 


0.776  0.224 
0.775  0.225 


and 


p7  =  p6p 


0.775  0.225 
0.775  0.225 


It  is  interesting  to  note  that  for  each  column  the  probabilities  in  that  column 
converge  to  the  same  value  (to  three  decimal  places).  So  the  probability  that  it  will 
be  dry  in  a  week’s  time  is  0.775  and  the  probability  that  it  will  be  wet  is  0.225;  this 
is  so  whether  it  is  dry  or  wet  today.  Although,  in  this  model,  the  weather 
tomorrow  does  depend  on  the  weather  today,  the  weather  in  a  week’s  time  appears 
to  be  not  affected  by  whether  it  is  dry  or  wet  today.  □ 


Example  1.5  Rainfall  in  Tel  Aviv 

In  a  study  of  rainfall  in  Tel  Aviv,  Gabriel  and  Neumann  used  data  for  the  rainy 
period — December,  January  and  February — over  twenty-seven  years.  They  found 
that  a  two-state  Markov  chain,  using  relative  frequencies  for  transition 
probabilities,  gave  a  good  description  of  the  occurrence  of  wet  and  dry  days  during 
the  rainy  period.  The  data,  covering  2437  days,  are  summarized  in  Table  1.2. 


Table  1.2 


Weather  on  the  following  day 
Dry  Wet 

Total 

Weather  on  one  day 

Wet 

1049  350 

351  687 

1399 

1038 

Question  1.6 

(i)  Use  the  data  in  Table  1.2  to  find  the  transition  matrix  P  of  Gabriel  and 
Neumann’s  Markov  chain  model. 

(ii)  Find  the  probability  that  if  it  rains  on  1  January  it  will  rain  on 
1 1  January.  □ 

Example  1.6  Job  mobility 

The  way  in  which  the  male  wage-earners  in  a  family  change  in  occupational  status 
from  generation  to  generation  has  been  of  considerable  interest  to  sociologists  for 
many  years.  In  1954,  Glass  and  Hall  obtained  data  on  the  occupational  status  of 
male  residents  of  England  and  Wales  and  on  the  occupational  status  of  their 
fathers.  The  matrix  in  Table  1.3  gives  the  transition  probabilities  estimated  from 
these  data  for  a  Markov  chain  model  with  three  states,  based  on  the  three 
occupational  levels  below. 

Level  1:  Upper  (professional,  high  administrative,  managerial,  executive) 
Level  2:  Middle  (higher-grade  supervisory  and  non-manual, 
skilled  manual  and  non-manual) 

Level  3:  Lower  (semi-skilled  manual,  unskilled  manual) 

Table  1.3 

Man’s  occupational  status 
1  2  3 


Father’s 

1 

0.45 

0.48 

0.07 

occupational 

2 

0.05 

0.70 

0.25 

status 

3 

0.01 

0.50 

0.49 

For  example,  p23  =  0.25  is  the  probability  that  the  son  of  a  man  in  the  middle- 
level  occupational  group  is  in  the  lower  group. 


Notice  that,  in  order  to  find  P7,  it  is 
not  necessary  to  calculate  all  the 
powers  of  P  up  to  P7,  but  simply  to 
obtain  P2,  P4,  P6  and  P7. 


K.  R.  Gabriel  and  J.  Neumann, 

‘A  Markov  chain  model  for  daily 
rainfall  in  Tel  Aviv’,  Quarterly 
Journal  of  the  Royal  Meteorological 
Society,  vol.  88  (1962)  pp.  90-5. 


D.  V.  Glass  and  J.  R.  Hall,  ‘A  study 
of  intergeneration  changes  in  status’, 
in  Social  Mobility  in  Britain 
(The  Free  Press,  1954)  pp.  177-241. 


To  fix  ideas,  you  could  regard  the 
underlying  random  process 
{X„;  n  =  0, 1, 2,  . . .}  as  describing  the 
occupational  status  of  males  in 
families  for  which  there  is  one  male 
member  in  each  generation.  The 
times  of  observation  n  =  0, 1, 2,  . . . 
are  generations  (within  families)  and 
the  values  of  X„  correspond  to  the 
three  occupational  levels. 
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Question  1.7 

(i)  What  assumptions  about  changes  in  occupational  status  are  made  by  using  a 
Markov  chain  model?  (That  is,  interpret  Conditions  (1)  and  (2)  for  this 
example.) 

(ii)  Find  the  probability  that  the  grandson  of  a  man  in  the  lower-level 
occupational  group  has  upper-level  occupational  status.  □ 


In  Examples  1.4  and  1.5,  powers  of  the  transition  matrix  P  were  calculated  to  help 
predict  the  weather  a  number  of  days  ahead.  For  the  Melbourne  model,  it  was 
found  that,  to  three  decimal  places,  the  rows  of  P7  were  equal,  so  that  any 
prediction  for  a  week  ahead  or  further  is  not  affected  by  today’s  weather;  so,  for 
example,  you  could  not  use  this  model  to  increase  your  chances  of  picking  a  fine 
day  for  an  outing,  if  the  outing  were  still  a  week  or  more  away.  In  the  Tel  Aviv 
example,  the  rows  of  P10  are  approximately  equal,  so  any  day  ten  or  more  days 
ahead  is  as  likely  to  be  wet  if  it  is  dry  today  as  it  is  if  it  is  wet  today.  In  these  two 
cases,  as  successive  powers  of  P  are  calculated,  the  probabilities  in  each  row 
appear  to  approach  limiting  values;  and  these  values  are  the  same  for  both  rows. 
So,  for  example,  irrespective  of  the  weather  on  day  0,  in  the  long  run  each 
Melbourne  day  will  be  dry  with  probability  0.775  and  wet  with  probability  0.225. 


In  Example  1.6  too,  it  appears  that  the  rows  of  P"  approach  a  limiting 
distribution;  direct  calculation  of  powers  leads  to 


pi°  _ 


'0.062  0.624  0.314' 

0.062  0.624  0.314 
0.062  0.624  0.3 14_, 

But  does  a  limiting  distribution  always  exist?  And  is  there  a  quicker  way  of  finding 
it  than  by  calculating  powers  of  P?  These  are  two  of  the  questions  that  are 
investigated  in  the  next  section,  where  the  long-run  behaviour  of  Markov  chains  is 
discussed. 


2  The  long-run  behaviour  of  Markov  chains 


So  far  we  have  considered  only  the  transition  probabilities  of  a  Markov  chain; 
that  is,  given  that  state  i  is  occupied  at  a  certain  time,  we  have  found  the 
probability,  p\f,  that  state  j  is  occupied  n  steps  later.  But  what  happens  if  the 
initial  state  is  not  given  but  itself  has  a  probability  distribution?  In  this  case,  what 
is  the  probability  of  being  in  state  j  after  n  transitions?  In  Subsection  2.1,  notation 
is  introduced  for  the  distribution  of  the  initial  state  of  a  Markov  chain  and  for  the 
absolute  (or  unconditional)  probability  of  a  state  being  occupied  after  n  steps;  then 
the  relationship  connecting  these  probabilities  and  the  transition  probabilities  is 
established.  The  discussion  in  Subsections  2.2  and  2.3,  of  the  behaviour  of  a 
Markov  chain  when  it  has  been  running  for  a  long  time,  leads  to  the  definition  of 
two  types  of  distribution  associated  with  Markov  chains:  limiting  distributions  and 
stationary  distributions. 


2.1  Absolute  probabilities 

The  probability  that  a  Markov  chain  is  initially  in  state  i  is  denoted  by  a,-;  that  is, 
=  P{X  0  =  i),  i  e  S. 

It  is  often  convenient  to  write  these  initial  probabilities  in  a  row  vector 

a|0)  =  [a0  ax  a2  ...].  The  absolute  probability  that  the  Markov  chain  is  in  state  j 

after  n  (>  1)  steps  is  denoted  by  ajn);  that  is, 

<  =  P(Xn  —  j). 

The  row  vector  a(n)  =  [a(0n)  a\n)  a(2n)  . . .]  gives  the  distribution  of  the  state  of  the 
Markov  chain  occupied  at  time  n.  To  find  a  relationship  connecting  these 


A  row  vector  is  written  with  square 
brackets  in  order  that  it  and  its 
representation  as  a  row  matrix  are 
identical. 
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(absolute)  probabilities  and  the  initial  probabilities,  we  shall  apply  the  Theorem  of 
Total  Probability  to  a =  P{X„  =  ;'),  conditioning  on  X0.  Thus,  for  jeS, 

Ptt  =i)=  I  P(K  =  i  |  x0  =  i)  P(x0  =  i). 

ieS 

That  is,  for  each  j  e  S,  in  terms  of  the  new  notation, 
af  =  I  ptfa,  =  I  fliP*?. 

ieS  ieS 

The  right-hand  side  is  the  jth  entry  in  the  product  of  the  row  vector  a(0)  and  the 
matrix  P(n)  =  P",  where  P  is  the  transition  matrix.  Thus,  the  above  expressions  can 
be  combined  into  matrix  form: 

a(n)  =  a(0)P",  n  >  1.  (2.1) 

A  formula  for  calculating  a<n+1),  the  absolute  probability  distribution  after  ( n  +  1) 
steps,  in  terms  of  that  distribution  after  n  steps  and  P,  is  the  subject  of  the 
following  question. 

Question  2.1  Show  that,  for  n  >  0, 

a("+i)  =  a(")P,  (2.2) 

by  following  an  argument  like  that  above  but  conditioning  on  Xn.  □ 

Example  2.1  Brand-switching 

A  supermarket  sells  two  brands  of  baked  beans,  brands  A  and  B.  A  student 
purchases  one  of  these  brands  each  day;  if  his  choice  of  brand  on  day  n  is  the  only 
factor  affecting  his  choice  of  brand  on  day  n  +  1,  then  the  sequence  of  brand 
choices  from  day  to  day  is  a  two-state  Markov  chain  with  states  0  (brand  A)  and  1 
(brand  B).  We  shall  suppose  it  is  known  that  an  individual  customer’s  brand 
choice  changes  from  day  to  day  as  a  Markov  chain  with  transition  probabilities 
given  by  the  matrix 

0  1 

p  _  0  j  )  These  probabilities  might  have  been 

1  |  i  estimated  from  survey  data. 

and  that  these  probabilities  are  the  same  for  all  customers. 

If  the  student  initially  has  no  preference  for  a  particular  brand,  then  the  initial 
state  of  the  Markov  chain  of  his  choices  has  distribution 

a(0)  =  [fl0  fll]  =  [*  *]. 

Using  Equation  (2.1)  with  n  =  1  (or  Equation  (2.2)  with  n  =  0),  the  probabilities 
that  he  will  choose  brand  A  or  brand  B  on  the  next  day  are  given  by 

a(1)  =  a(0)P, 

so  that 

L41’  ai1’]  “  Li  H  \  \ 

=  Cf  f]  =  [0.625  0.375]. 

Hence,  the  probability  that  the  student  will  buy  brand  A  on  day  1  is  f;  and  he  will 
buy  brand  B  with  probability  |.  Using  Equation  (2.2)  with  n  =  1  gives 

a(2)  =  a(l)p, 

so  that 

[«?>  an=[H][l  \ 

_4  4_ 

=  Ilf  J0  -  [0.594  0.406]. 

So  he  will  buy  brand  A  on  day  2  with  probability  0.594  approximately.  □ 
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Question  2.2  Find  the  probabilities  (correct  to  three  decimal  places)  that  the 
student  will  buy  brands  A  and  B  on  each  of  days  3,  4  and  5.  □ 

In  this  example,  a  Markov  chain  model  has  been  used  to  find  the  probability  that 
an  individual  student  will  buy  a  particular  brand  on  a  particular  day;  the 
distribution  of  the  initial  state  contains  the  probabilities  that  this  student  will  buy 
brand  A  or  brand  B  on  day  0,  and  the  absolute  probabilities  are  the  probabilities 
that  he  will  buy  them  on  succeeding  days.  However,  there  is  another  way  that  this 
model  can  be  used;  since  all  customers  behave  like  this  student,  the  supermarket 
manager  could  use  the  model  to  predict  how  much  of  each  brand  will  be  sold  on 
future  days.  To  do  this  the  initial  distribution  must  be  interpreted  in  a  different 
way,  with  the  probabilities  a0  and  al  being  the  proportions  of  customers  buying 
brand  A  and  brand  B,  respectively,  on  day  0;  so  if  a0  =  ax  =  initially  50%  of 
customers  buy  each  brand.  Then  the  absolute  probabilities  represent  the  predicted 
proportions  of  customers  buying  brand  A  and  brand  B  on  later  days.  For  example, 
after  5  days  approximately  60%  of  customers  buying  baked  beans  from  the 
supermarket  are  buying  brand  A. 

Question  2.3  -  In  Example  1.6  transition  probabilities  are  given  for  changes  in 
occupational  status  between  two  consecutive  generations  within  families.  In  the 
light  of  the  preceding  discussion,  how  would  you  interpret  an  initial  distribution 
and  the  resulting  absolute  probabilities  for  this  model?  □ 


2.2  Limiting  distributions 

At  the  end  of  Section  1,  you  saw  that,  in  each  of  Examples  1.4  (Melbourne 
weather),  1.5  (Rainfall  in  Tel  Aviv)  and  1.6  (Job  mobility),  the  n-step  transition 
probabilities  appear  to  approach  limiting  values,  independent  of  the  initial  state. 
This  is  not  so  for  every  Markov  chain.  However,  many  Markov  chains  do  achieve 
a  state  of  equilibrium  in  the  sense  that  the  conditional  and  unconditional 
probabilities  (p\f  and  respectively)  of  observing  the  chain  in  the  different  states 
eventually  cease  to  depend  on  the  initial  state  of  the  chain  or  on  the  length  of  time 
since  the  process  started.  An  illustration  of  this  is  given  by  the  two-state  Markov 
chain  model  for  brand-switching  in  Example  2.1.  Table  2.1  shows  the  absolute 
probabilities  of  brand  choice  (obtained  in  that  example  and  Question  2.2)  for  the 
first  five  days  when  a(0)  =  £],  and  Table  2.2  gives  the  matrices  of  n-step 

transition  probabilities  for  n  up  to  6. 


Table  2.1  Absolute  (or  unconditional)  probabilities  (correct  to  three 
decimal  places)  of  brand  choice 


Day 

0 

1 

2 

3 

4 

5 

atf  =  P(Xn 

—  0) 

0.500 

0.625 

0.594 

0.602 

0.600 

0.600 

a<">  =  P(Xn 

=  1) 

0.500 

0.375 

0.406 

0.398 

0.400 

0.400 

Table  2.2  The  matrices  of  w-step  transition  probabilities  (correct  to  three  decimal  places)  for  changes  in  brand  choice 


1 


P" 


0.500 

0.750 


0.500 

0.250 


0.625 

0.562 


0.375 

0.438 


0.594  0.406 
0.609  0.391 


0.602  0.398 
0.598  0.402 


0.600  0.400 
0.600  0.400 


0.600  0.400 
0.600  0.400 


Looking  at  the  matrices  in  Table  2.2,  you  can  see  that,  for  each  i  and  j,  the 
transition  probabilities,  p|"),  settle  down  to  a  limiting  value  as  n  increases;  and  this 

limiting  value  is  independent  of  i,  the  initial  state  of  the  Markov  chain.  And  from  i  specifies  the  row  of  P" 

Table  2.1,  it  is  clear  that,  for  each  state  j,  the  absolute  probabilities,  a{"\  also 

approach  this  limiting  value.  When,  in  the  limit  as  n  ->  oo,  a  Markov  chain 

displays  such  equilibrium  behaviour,  the  Markov  chain  is  said  to  be  in 

probabilistic  (or  stochastic)  equilibrium.  The  limiting  value  is  denoted  by  Tij,  j  e  S. 


Not  all  Markov  chains  ‘settle  down’  in  this  way.  However,  if  a  Markov  chain  does 
achieve  probabilistic  equilibrium  then,  for  all  sufficiently  large  n,  regardless  of  the 
state  in  which  the  process  begins,  the  probability  of  observing  it  in  state  j  at  time  n 
has  approximately  the  value  Uj.  That  is,  for  all  i  e  S, 

Pu]  71  i  as  n  ->  co, 
and,  for  all  sufficiently  large  n , 

p'?  = 

Moreover,  for  each  state  j, 
aW  — ►  Uj  as  n  ->  co 
and  for  all  sufficiently  large  n. 


^  7 Z:. 


The  limiting  value  Uj  of  the  n-step  transition  probabilities  p-"*,  which  is  common  to 
all  initial  states  i,  is  called  the  limiting  probability  of  state  j.  Note  that  since 
0  <  pff  <  1  for  all  i,  j  and  n,  it  follows  that  0  <  Uj  <  1.  When  the  sum  £  Uj  is 

jeS 


equal  to  1,  {ny.  jeS}  constitutes  the  probability  distribution  that  assigns  the 
probability  Uj  to  state  j;  it  is  called  the  limiting  distribution  of  the  Markov  chain.  If 
this  distribution  is  known,  then  we  can  eliminate  most  of  the  hard  work  involved 
in  calculating  n-step  transition  probabilities  for  large  values  of  n,  since,  for 
example,  for  a  two-state  Markov  chain,  Pn  is  given  approximately  by 


More  generally,  for  a  Markov  chain  with  states  0, 1,  ...,  m,  P"  is  given 
approximately  by 


7T0 

*1  ■ 

..  K, 

7l0 

nl  . 

..  n, 

n0 

n1  .. 

..  71, 

Note  that  the  symbol  7t7-  is  not 
reserved  for  the  limiting  probability 
of  state  j;  in  particular,  see 
Example  2.4. 


Interpretation  of  the  limiting  distribution 

The  primary  interpretation  of  the  limiting  probability  tcj  is  as  the  probability  of 
finding  the  process  in  state  j  after  it  has  been  running  for  a  long  time.  However, 
there  is  a  second  interpretation,,  which  is  important  in  many  applications  of 
Markov  chain  models,  in  which  nj  is  the  long-run  mean  proportion  of  time  that 
the  Markov  chain  spends  in  state  j.  To  see  that  this  is  a  valid  interpretation,  we 
shall  first  find  the  expected  number  of  visits  the  Markov  chain  makes  to  state  j  in 
the  first  N  transitions. 

For  fixed  j,  the  indicator  of  the  event  [ Xn  =  ;']  will  be  denoted  by  /„;  that  is,  /„ 
takes  the  value  1  if  the  Markov  chain  is  in  state  j  at  time  n,  and  0  if  it  is  not  in 
state  j  : 


N 

Then  £  /„  is  the  total  number  of  visits  to  state  j  in  the  first  N  transitions;  and, 

n  =  1 

given  that  the  Markov  chain  starts  in  state  i,  the  expected  number  of  visits  to  state 
j  in  this  time  is 

I  /„|X0  =  i)=  £  E(/„|*0  =  i) 

=  1  J  n  = 1 

=  £  1  x  />(/„  =  1  |X0  =  i)  +  o  x  P(I.  =  0|X„  =  i) 

n  =  1 
N 

=  Yj  P(Xn  =j\X0  =  0>  by  definition  of 

n=  1 

=  £  Pi?- 

n  =  1 


l  if  xn=j 
0  if  xn*j. 
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1  N 

So  —  X  Pi?  is  the  expected  proportion  of  this  time  (N)  spent  in  state  j.  But  a 
w  n  =  1 

simple  result  from  analysis  states  that  if  a  sequence  of  real  numbers  bl,b2,  ... 
converges  to  a  limit  fr,  then  the  sequence  of  averages  bt,  (by  +  b2)/ 2, 

(hi  +  b2  +  b3)/ 3,  ...  also  converges  to  h;  that  is, 

1  N 

lim  T7  I  bn  =  b. 

'v-oo  n£x 


If  this  result  is  applied  to  the  sequence  {pW},  which  converges  to  nJt  then  we  have 


1  N 

lim  I  Pi? 


That  is,  7r j  is  the  long-run  mean  proportion  of  time  spent  in  state  j  by  the  Markov 
chain;  and  this  is  independent  of  the  starting  state  i. 


For  the  student  in  Example  2.1,  we  can  deduce  that  over  a  long  period  of  time,  k0 
is  the  long-run  mean  proportion  of  days  on  which  he  buys  brand  A  baked  beans. 
And  for  Example  1.4  (Melbourne  weather),  n0  and  nx  are  the  long-run  mean 
proportions  of  days  (between  January  and  March)  that  are  dry  and  wet 
respectively. 


Question  2.4  Express  in  words  the  two  possible  interpretations  of  the  limiting 
probabilities  for  the  job  mobility  Markov  chain  model  of  Example  1.6.  □ 


2.3  Limiting  distributions  and  stationary  distributions 


Example  2.1  Brand-switching,  continued 

From  Table  2.2  it  appears  that,  for  the  brand-switching  example,  n0  =  0.600  and 
7rj  =  0.400,  correct  to  three  decimal  places.  We  shall  now  show  that  0.6  and  0.4  are 
the  exact  values  of  the  limiting  probabilities;  we  shall  do  this  by  finding  an 
expression  for  pftl  and  letting  n  increase  indefinitely.  You  are  asked  to  do  the  first 
part  of  the  working  in  the  next  question. 

Question  2.5  Use  the  relationship  P(n+1)  =  P(n)P  jn  the  brand-switching  example 
to  show  that,  for  n  >  1, 

Poo+1)  =  I  -  *  Poo-  □  (2.3) 

Equation  (2.3)  is  a  first-order  recurrence  relation  of  the  form 

un+1=cun  +  d,  for  n  >  1.  (2.4) 

This  can  be  solved  by  repeated  substitution: 
un  =  cun_!  +  d 

=  c(cun  _  2  +  d)  +  d 
=  d  +  cd  +  c2(cu„_  3  +  d) 


=  d  +  cd  +  czd  +  ...  +  cn~ld  +  c"uq. 
Provided  c  #  1,  this  can  be  written 
d( 1  -  cn)  n 

“"  =  “rrr  +  c"“0- 

Comparing  Equations  (2.3)  and  (2.4), 

u„  =  Poo,  d  =  l  and  c=-£. 

So  as  n  -*■  oo,  cn  ->  0,  hence 
d 


POO  ~  Un 


1  —  C 


=  0.6. 


In  a  similar  way,  it  can  be  shown  that  p%  ->  0.6  as  n  ->  oo;  and  using  the  fact  that, 
for  each  n,  +  p(0"}  =  1  and  =  1,  it  follows  that  p(0"}  -»  0.4  and  p^\  -+  0.4 

as  n  ->  oo.  Hence,  for  this  example,  the  limiting  probabilities  exist  and  their  values 
are  0.6  and  0.4;  that  is,  n0  =  0.6  and  7rx  =  0.4.  □ 


Thus,  the  proviso  c  =£  1  is  satisfied. 
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In  the  case  of  this  two-state  Markov  chain  model  it  has  been  shown  that  (for 
Uj  =  0, 1)  the  rc-step  transition  probabilities  pff  converge  to  a  limiting  probability 
7tj.  However,  finding  the  limiting  probability  involved  obtaining  and  solving  a 
recurrence  relation;  even  this  simple  example  involved  a  fair  amount  of  work. 
Furthermore,  the  method  used  cannot  be  generalized  for  use  with  a  Markov  chain 
with  more  than  two  states;  another  way  of  finding  the  limiting  distribution  is 
required.  We  shall  approach  the  problem  of  finding  such  a  method  by  assuming 
that  the  rc-step  transition  probabilities  pj”'  tend  to  some  limiting  value  Uj  as  n  ->  oo, 
and  investigating  what  properties  Kj  must  have.  In  Section  3  we  shall  return  to  the 
question  of  whether  or  not  a  limiting  distribution  exists. 

Suppose  that  a  Markov  chain,  with  state  space  S’  =  (0, 1,  ...,  m}  and  transition 
matrix  P,  has  a  limiting  distribution  n  =  [7t0  n x  ...  nm];  that  is,  for  all  ieS ,  and 
for  each  j  e  S, 

lim  p\f  =  n j. 

n~*  co  J  J 

From  the  Chapman-Kolmogorov  equations  (1.8)  we  have,  for  n  >  1, 

m 

Pu  +  1>  ~  Z  P*Pkj-  (2.5) 

fc  =  0 

But  p$  -»  nk  and  p\j  +  1>  -*■  itj  as  n  -*•  co,  so,  letting  n  -*  co  in  Equation  (2.5)  gives, 
for  j  =  0, 1,  ...,  m, 

m 

nj  =  Z  nkPkj  (2.6) 

k  =  0 

or,  equivalently,  in  terms  of  matrices, 

n  =  nP.  (2.7) 

Furthermore,  as  we  have  seen, 

nj>  0  for  all  j  eS,  (2.8) 

m 

and,  since  Z  Pij*  =  1  f°r  1  an<^  n,  letting  n  ->  co  gives 
J=  o 

m 

Z  "j  =  !•  (2.9) 

7  =  0 

So,  if  it  exists,  the  limiting  distribution  n  of  a  finite  Markov  chain  satisfies 
Conditions  (2.7),  (2.8)  and  (2.9);  and,  by  finding  the  solution  of  Equation  (2.7) 
satisfying  Conditions  (2.8)  and  (2.9),  we  can  find  this  limiting  distribution.  We  shall 
do  this  first  for  the  brand-switching  Markov  chain. 


Example  2.1,  continued 


For  the  Markov  chain  model  for  brand-switching  with  transition  matrix 


we  require  a  solution  n  =  [>0  n{]  of  the  equation 

|>o  7 tj  =  O0  7 rj  |j  | 


(2.10) 


such  that  7r0  >  0,  >0  and  7i0  +  =  1.  There  are  two  equations  in  (2.10);  these 

are  /  \  x  ,  /  O 


nv  —  Oo  +  kni  ■ 

However,  one  of  these  is  redundant,  since  adding  them  leads  to  n0  +  nx  —  n0  +  nx; 
that  is,  the  equations  are  linearly  dependent,  and  we  must  discard  one  of  them,  say 
the  second.  Then  we  must  find  a  non-negative  solution  (Condition  (2.8))  of  the 
other  equation  satisfying  n0  +  =  1;  that  is,  we  must  solve  the  simultaneous 

equations 

TIq  =  277o  +  4^1  and  TLq  +  7Tj  =  1. 


It  does  not  matter  which  of  the  two 
equations  we  discard. 


This  leads  to  n0  =  0.6  and  =  0.4,  as  before.  □ 
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Example  2.2  Pushkin’s  ‘Eugene  Onegin’ 

Markov  sought  the  probability  that  a  letter  randomly  chosen  from  a  Russian  text 
would  turn  out  to  be  a  vowel.  He  took  a  sequence  of  20000  letters  from  Pushkin’s 
‘Eugene  Onegin’  and  found  that,  for  this  text,  the  probability  of  occurrence  of  a 
vowel  after  a  vowel  is  0.128  and  the  probability  of  a  vowel  after  a  consonant  is 
0.663.  (These  were  estimated  from  relative  frequencies.)  He  asserted  that  this 
sequence  can  be  modelled  by  a  simple  Markov  chain  with  two  states,  which  we 
shall  label  0  and  1,  corresponding  to  a  vowel  and  a  consonant,  respectively;  this 
Markov  chain  has  transition  matrix 
0  1 

0  I" 0.128  0.872“ 

“  1  j_0.663  0.337' 

If  this  is  a  satisfactory  model,  then  the  limiting  probability  n0  should  be 
approximately  equal  to  the  fraction  of  letters  in  the  (whole)  text  that  are  vowels. 
The  limiting  distribution  n  =  [ti0  must  satisfy  Conditions  (2.7),  (2.8)  and  (2.9); 
that  is,  Kq  and  71!  must  be  non-negative  and,  from  the  first  equation  in 
Condition  (2.7), 

n0  =  0.128  n0  +  0.663  n1, 
which  simplifies  to 

0.872  7i0  =  0.663  tti 
and,  from  Condition  (2.9), 

710  +  71!  =  1. 

Thus,  n0  is  the  solution  of 

0.872  tt0  =  0.663(1  -  7t0). 

So,  to  three  decimal  places,  7t0  =  0.432.  In  fact,  Markov  found  that  this  is  very 
close  to  the  frequency  of  occurrence  of  vowels  in  the  text  of  ‘Eugene  Onegin’.  □ 


Question  2.6  Use  Conditions  (2.7),  (2.8)  and  (2.9)  to  find  the  limiting  distribution 
of  a  two-state  Markov  chain  with  transition  matrix 


where  0  <  a  <  1,  0  <  /?  <  1  and  a  and  /?  are  not  both  equal  to  zero  and  not  both 
equal  to  one.  □ 


Question  2.7  Use  the  results  of  Question  2.6  to  find  the  limiting  distribution  for 
each  of  the  Markov  chain  models  for  weather  described  in  Section  1 : 

(i)  Example  1.4,  Melbourne  weather; 

(ii)  Example  1.5,  Rainfall  in  Tel  Aviv.  □ 


Although  Conditions  (2.7),  (2.8)  and  (2.9)  were  derived  for  any  finite  Markov  chain 
that  has  a  limiting  distribution,  so  far  we  have  used  them  only  for  a  two-state 
Markov  chain.  We  shall  now  use  them  to  find  the  limiting  distribution  for  the 
Markov  chain  model,  described  in  Example  1.6,  for  changes  in  occupational  status 
between  generations. 
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Example  2.3  Job  mobility 

The  transition  matrix  P  for  changes  in  occupational  status  is 
1  1" 0.45  0.48  0.07~ 

P  =  2  0.05  0.70  0.25  . 

3  |_0.01  0-50  0.49_ 

Assuming  that  this  Markov  chain  has  a  limiting  distribution,  n  =  [nx  n2  7t3],  it 
must  satisfy  Conditions  (2.7),  (2.8)  and  (2.9).  Condition  (2.7)  gives  three  equations 
for  nx,  ti2  and  7t3,  one  of  which  is  redundant.  Discarding  the  third  equation,  we 
have 


nx  =  0.45  7r !  +  0.05  n2  +  0.01  n3, 

(2.11) 

n2  =  0.48  7Tj  +  0.70  7i2  +  0.50  n3. 

(2.12) 

Condition  (2.9)  is 

7^!  -f-  7l2  -f-  7Z3  =  1. 

(2.13) 

This  gives  three  simultaneous  equations  for  nx,  n2  and  n3. 
to  substitute  for  n3  in  Equations  (2.1 1)  and  (2.12)  gives 

Using  Equation  (2.13) 

71 1  =  0.45  71!  +  0.05  7E2  +  0.01(1  —  7Ti  —  7T2), 

7t2  =  0.48  7i  1  +  0.70  tt2  +  0.50(1  -  nx  -  n2), 

that  is, 

0.56 Tii  -  0.04 tt2  =  0.01, 

(2.14) 

0.02  7c  1  +  0.80  tt2  =  0.50. 

(2.15) 

Multiplying  Equation  (2.14)  by  20  gives 

11.20  Tii  -  0.80  te2  =0.20. 

(2.16) 

Then  adding  Equations  (2.15)  and  (2.16)  leads  to  11.2271!  =  0.70,  so  that 
71 1  =  0.062  39.  Equation  (2.14)  gives  n2  —  0.623  46;  and  from  Equation  (2.13),  we 
have  7i3  =  1  -  nx  —  7r2  =  0.314  15.  So,  correct  to  three  decimal  places,  the  limiting 
distribution  is 

7i !  =  0.062,  n2  =  0.623  and  n2  =  0.314. 

Hence,  in  the  long  run,  the  proportions  of  male  wage-earners  in  each  of  the  three 
occupational  levels  will  be  approximately  0.062,  0.623  and  0.314.  □ 


Question  2.8  Another  study  of  change  in  occupational  status  suggested  that  the 
transition  matrix  for  the  Markov  chain  model  should  be 
/  [0.5  0.4  0.1" 

P  =  2  0.2  0.6  0.2  . 

3  [_0.1  0.2  0.7_ 

Under  this  model,  in  the  long  run  what  proportion  of  male  wage-earners  will  be  in 
the  upper  occupational  level?  □ 


Although,  if  it  exists,  the  limiting  distribution  must  satisfy  Conditions  (2.7),  (2.8) 
and  (2.9),  care  is  needed  when  applying  this  method  as  not  every  distribution 
satisfying  Condition  (2.7)  is  a  limiting  distribution.  This  is  illustrated  in  the  next 
example. 


Example  2.4 

For  the  Markov  chain  with  transition  matrix  P  =  ^ 


0 

1 


1 

0 


the  distribution 


7i  =  i>0  71 1]  =  [2  2]  satisfies  Conditions  (2.7)-(2.9),  since 

h  i]  =  ti  u  1  I, , 

n0  >  0,  nx  >  0  and  7t0  +  7il  =  1.  But  P2m  = 


1  0 
0  1 


and  P2m  +  1  = 


0  1 
1  0 


for 


m  =  1,2,  ...  ;  so  for  any  fixed  i  and  j  the  sequence  of  probabilities  {pW}  oscillates 
between  0  and  1  and  so  does  not  approach  a  limit.  Thus,  neither  \  in  n  =  [£  is 
a  limiting  probability  and  the  Markov  chain  does  not  have  a  limiting 
distribution.  □ 


From  here  on,  states  are  indicated 
by  row  only. 


Adding  the  three  equations  leads  to 

7C]  +  U2  +  71 3  =  7C  j  +  7l2  +  7I3. 
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Also,  as  the  next  example  shows,  for  some  Markov  chains  there  is  more  than  one 
non-negative  solution  of  Equations  (2.7)  and  (2.9). 


Example  2.5 

For  the  Markov  chain  with  transition  matrix 
0  [  1  0  O' 

p  —  i  iii 
r  1  3  3  3  > 

2  [0  0  1 

n  =  Cl  0  -}]  and  n  =  0  |]  are  both  examples  of  distributions  satisfying 

Condition  (2.7)  and  there  are  many  more — an  infinite  number,  in  fact.  So 
Equations  (2.7)  and  (2.9)  do  not  have  a  unique  non-negative  solution.  □ 


Each  limiting  distribution  is  a 
stationary  distribution. 

a1"1  is  the  distribution  of  the  state 
occupied  at  time  n. 


The  distributions  n  in  Examples  2.4  and  2.5  are  stationary  distributions  but  they 
are  not  limiting  distributions  (for  the  Markov  chains  with  the  given  transition 
matrices).  The  conditions  for  a  stationary  distribution  to  be  a  limiting  distribution 
are  the  subject  of  Subsection  3.2. 

So  far  in  this  subsection  the  basic  properties  of  limiting  probabilities  have  been 
obtained  and  stationary  distributions  have  been  introduced  only  for  Markov 
chains  with  a  finite  state  space;  however,  similar  results  hold  for  infinite  Markov 
chains,  although  the  derivation  is  a  little  more  complicated.  In  Section  5,  you  will 
need  to  find  a  stationary  distribution  for  a  Markov  chain  with  an  infinite  state 
space;  so  this  section  closes  by  summarizing  the  important  points  for  both  finite 
and  infinite  Markov  chains  and  discussing  briefly  how  and  why  the  conditions 
satisfied  by  the  limiting  probabilities  differ  when  the  state  space  is  infinite. 


Any  distribution  n  which  satisfies  Conditions  (2.7),  (2.8)  and  (2.9)  is  called  a 
stationary  distribution;  this  name  is  used  because  of  a  property  held  by  all 
stationary  distributions.  Suppose  n  is  a  stationary  distribution  for  a  Markov  chain 
and  that  the  initial  distribution,  a(0),  is  chosen  to  be  n.  Then,  using  Relation  (2.2), 
a(n+1)  =  a(,,)P,  to  find  a(1),  we  have 

a(1)  =  nP  =  71, 

by  Condition  (2.7).  Similarly,  applying  (2.2)  and  (2.7)  successively  gives  a<M)  =  n  for 
all  n.  Thus,  if  the  distribution  of  the  initial  state  is  chosen  to  be  a  stationary 
distribution,  then  each  subsequent  state  has  the  same  distribution;  that  is,  the 
distribution  of  the  state  occupied  is  stationary  over  time. 


Limiting  probabilities  and  stationary  distributions 


If  a  Markov  chain  is  such  that,  for  each  jeS,  lim  ptf  =  nh  then 

n-*  oo  1  J 

n  =  [7r0  nx  ...]  satisfies  the  following  two  conditions: 

(1)  n  =  71P; 

(2)  Tij  >  0  for  each  state  jeS. 

If  the  state  space  S  is  finite,  then  n  will  also  satisfy  a  third  condition: 

(3)  Z  71  i  =  L 

JeS 

Any  distribution  n  (including  those  for  which  lim  p)"*  =  n j)  satisfying  all  three 

conditions  is  called  a  stationary  distribution  for  the  Markov  chain.  (This 
definition  applies  also  to  infinite  Markov  chains.) 
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When  dealing  with  Markov  chains  with  an  infinite  state  space,  more  care  is 
needed.  The  third  condition  is  not  necessarily  satisfied  by  the  limiting  probabilities 
in  this  case,  since  for  an  infinite  state  space  we  cannot  deduce  Y  ns  =  1  from  the 
fact  that  Yj  P(u  =  1  for  ad  n •  This  is  so  because  jeS 

jeS 

lim  Y  Pij]  and  Z  lim  pj? 

JeS"-00 

may  not  be  equal;  when  the  summation  is  over  an  infinite  number  of  terms,  in 
general,  we  cannot  interchange  the  order  of  the  two  operations  of  summing  and 
taking  a  limit  as  n  ->  oo. 


An  example  of  a  Markov  chain  which  has  limiting  probabilities  but  for  which 
Condition  (3)  is  not  met  is  given  by  an  unrestricted  simple  random  walk  on  the 
line.  From  Formula  (4.1)  of  Unit  5,  it  follows  that,  for  an  unrestricted  simple 
random  walk  with  p  +  q  =  1,  for  n  >  1, 


U2n  =  POO  = 


so  that,  for  all  states  j,  pgn)  =  p(020n).  This  probability  tends  to  0  as  n  ->  oo.  Also 
from  Formula  (4.1)  of  Unit  5, 

Pnn  +  1)  =  Poo+1)  =  0  for  all  n>0. 


so  that  p}?  -»  0  as  n  ->  oo.  In  fact,  for  all  states  i  and  j,  pff  ->  0  as  n  -»  oo,  so  the 
limiting  probabilities  are  all  equal  to  zero,  and  hence  do  not  add  up  to  1.  So  this 
Markov  chain  has  limiting  probabilities,  but  they  do  not  form  a  distribution. 


3  Communicating  classes 


In  Section  2  the  ideas  of  limiting  probabilities  and  stationary  distributions  were 
introduced;  and  for  Markov  chains  in  general,  the  following  two  questions  are  of 
interest.  Under  what  conditions  does  a  Markov  chain  have  a  unique  stationary 
distribution?  And,  after  the  chain  has  been  running  for  a  long  time,  is  the 
distribution  of  the  state  occupied  approximately  equal  to  this  stationary 
distribution?  That  is,  under  what  circumstances  does  a  Markov  chain  have  a 
limiting  distribution? 

In  this  section  these  two  questions  will  be  discussed;  but  in  order  to  make  progress 
we  need  to  look  more  closely  at  the  structure  of  a  Markov  chain  and  the 
properties  of  its  states. 


3.1  Accessibility  between  states 


State  j  is  said  to  be  accessible  from  state  i  if  p\f  >  0  for  some  integer  n  >  0.  In 
particular,  since  pjf*  =  P(X0  =  i|  X0  =  i)  =  1,  every  state  i  is  accessible  from  itself; 
and  a  state  j,  other  than  i,  is  accessible  from  i  if  it  is  possible  to  get  from  i  to  j  in  a 
finite  number  of  steps.  Two  states,  i  and  j ,  each  of  which  is  accessible  from  the 
other,  are  said  to  communicate,  and  this  is  written  i  <-> j.  When  deciding  which 
states  of  a  Markov  chain  communicate,  it  is  much  easier  to  work  from  a  diagram, 
such  as  the  one  in  the  following  example,  than  from  a  matrix  of  transition 
probabilities. 


Example  3.1 


The  Markov  chain  with  state  space  {0, 1,2,3}  and  transition  matrix 


P  = 


0 

1 

2 

3 


i  j  0  0 

!  I  o  o 

o  i  i  i 

O  4  2  4 

0  0  0  1 


can  be  represented  by  the  diagram  in  Figure  3.1,  in  which  each  positive  transition 
probability,  pu,  is  represented  by  an  arrow  from  i  to  j. 


pV  =  P(Xm+n=j\Xm  =  i) 


If  distinct  states  i  and  j  communicate, 
then  there  exist  integers  r  >  1  and 
s  >  1  such  that  ptf  >  0  and  pfi  >  0. 
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Figure  3.1 

From  the  diagram  it  is  easy  to  see  that  the  states  0  and  1  communicate  (()<-►  1); 
however,  although  states  0  and  1  are  accessible  from  state  2,  state  2  is  not 
accessible  from  either  0  or  1.  Similarly,  state  3  is  accessible  from  state  2  but  state  2 
is  not  accessible  from  state  3;  and  the  pairs  0  and  3,  1  and  3  are  each  composed  of 
mutually  inaccessible  states.  □ 


In  this  example,  0  <-»  0,  0  1,  1  <-»  1,  2 «-» 2  and  3  <->  3,  but  no  other  pair  of  states 

communicates.  So  the  state  space  {0, 1,2,3}  may  be  divided  into  non-overlapping 
subsets — (0, 1},  {2},  {3} — each  subset  consisting  of  all  states  which  communicate 
only  with  each  other.  In  fact,  for  any  Markov  chain,  the  state  space  may  be 
divided  into  non-overlapping  subsets  of  states  in  such  a  way  that  two  states  are  in 
the  same  subset  if  and  only  if  they  communicate.  These  subsets  of  communicating 
states  are  called  communicating  classes. 

Example  3.2 


Figure  3.2 


If  you  have  studied  equivalence 
relations,  then  you  may  recognize 
that  the  relation  ‘communicates 
with’  is  an  equivalence  relation  on 
the  state  space  and  the 
communicating  classes  are 
equivalence  classes. 


For  the  purpose  of  identifying 
communicating  states  it  is  not 
necessary  to  write  the  transition 
probabilities  on  the  diagram. 


Figure  3.2  is  a  diagram  representing  the  Markov  chain  with  transition  matrix 


P  = 


0 

1 

2 

3 


0 

0.3 

0.8 

0 


0.4 

0 

0.2 

0 


0.6  0 
0.7  0 
0  0 
1  0 


For  this  Markov  chain,  states  0,  1  and  2  communicate  but  state  3  is  inaccessible 
from  the  other  states.  The  communicating  classes  are  (0, 1,2}  and  {3}.  □ 


For  the  Markov  chains  used  to  model  rainfall  patterns  (Examples  1.4  and  1.5)  and 
job  mobility  (Example  1.6),  all  states  form  a  single  communicating  class.  A  Markov 
chain  in  which  all  the  states  communicate  is  said  to  be  irreducible;  in  this  case 
there  is  only  one  communicating  class,  the  state  space  itself.  If,  once  in  a  class  C,  it 
is  impossible  to  leave,  then  C  is  said  to  be  a  closed  class;  that  is,  C  is  closed  if 
Pij  =  0  whenever  z'eC  and  j$C.  An  irreducible  Markov  chain  has  only  one  class, 
and  this  is  closed.  But  in  a  Markov  chain  with  more  than  one  class,  some  classes 
may  be  closed  and  others  not;  in  Example  3.1,  {0, 1}  and  {3}  are  both  closed 
classes,  but  {2}  is  not  closed.  And  in  Example  3.2,  {0, 1,2}  is  closed  but  {3}  is  not 
closed. 


To  see  this,  draw  the  diagrams. 

It  is  common  practice  to  refer  to  a 
‘communicating  class’  simply  as  a 
‘class’.  From  here  on,  whenever  the 
term  ‘class’  is  used,  ‘communicating 
class’  is  to  be  understood. 


Question  3.1  Find  the  communicating  classes  of  the  Markov  chains  having  the 
following  transition  matrices,  and  decide  whether  or  not  each  class  is  closed. 


(i) 


0 

P  =  1 
2 


1 

1 

A 

0 

3 

4 

1 

4 

0 

0 

2 

1 

I 

0 

2 

1 

2 

0 

0 

0 

1 

(ii)  P  =  \ 

1 

3 

0 

1 

3 

0 

1 

3 

1 

4 

0 

3 

4 

3 

0 

0 

1 

2 

1 

2 

□ 


Question  3.2  List  the  communicating  classes  for  the  gambler’s  ruin  Markov  chain 
(Example  1.1),  and  say  whether  or  not  they  are  closed.  □ 
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The  uniqueness  of  stationary  distributions 

We  shall  now  return  to  the  question  of  whether  or  not  a  finite  Markov  chain  has  a 
unique  stationary  distribution.  We  shall  look  again  at  several  examples  of  Markov 
chains  that  do  have  a  unique  stationary  distribution  and  one  that  does  not;  and,  in 
each  example,  we  shall  note  how  many  communicating  classes  the  Markov  chain 
has  and  whether  or  not  the  classes  are  closed. 

The  Markov  chains  in  Example  1.4  (Melbourne  weather),  Example  1.5  (Rainfall  in 
Tel  Aviv)  and  Example  1.6  (Job  mobility)  are  all  irreducible— that  is,  they  each 
have  one  communicating  class,  which  is  closed;  and  for  each  chain  Conditions 
(2.7),  (2.8)  and  (2.9)  have  a  unique  solution,  so  each  of  these  chains  has  a  unique 
stationary  distribution.  On  the  other  hand,  the  Markov  chain  in  Example  2.5,  for 
which  the  transition  matrix  is 

1  0  o' 

i  i  3  > 

0  0  1 

has  two  closed  classes,  {0}  and  {2},  and  one  class,  {1},  which  is  not  closed;  and  it 
has  an  infinite  number  of  stationary  distributions.  In  fact,  the  uniqueness  or 
otherwise  of  a  stationary  distribution  depends  on  the  number  of  communicating 
classes  of  the  Markov  chain;  in  particular,  it  depends  on  the  number  of  closed 
classes. 

A  simple  criterion  for  the  existence  of  a  unique  stationary  distribution  for  a 
Markov  chain  is  given  by  the  following  theorem;  this  result  will  not  be  proved. 


Theorem  3.1 

Every  Markov  chain  with  a  finite  state  space  has  a  unique  stationary 
distribution  unless  the  chain  has  two  or  more  closed  communicating  classes. 


This  theorem  allows  us  to  decide  without  calculation  whether  or  not  a  Markov 
chain  has  a  unique  stationary  distribution.  For  example,  since  the  Markov  chains 
used  to  model  rainfall  patterns  and  job  mobility  are  irreducible,  they  each  have 
only  one  communicating  class,  so  we  can  deduce  from  the  theorem  that  they  each 
possess  a  unique  stationary  distribution.  But  the  Markov  chain  in  Example  3.1  has 
two  distinct  closed  classes,  (0, 1}  and  {3},  so,  although  the  Markov  chain  possesses 
a  stationary  distribution,  it  is  not  unique. 

Notice  that  the  theorem  states  that  the  stationary  distribution  is  unique  unless  the 
chain  has  two  or  more  closed  classes;  so  a  Markov  chain  which  has  two  or  more 
communicating  classes,  only  one  of  which  is  closed,  does  have  a  unique  stationary 
distribution.  For  example,  the  Markov  chain  in  Example  3.2  has  two 
communicating  classes:  (0, 1,2},  which  is  closed,  and  {3},  which  is  not  closed.  It 
has  only  one  closed  class,  so  it  has  a  unique  stationary  distribution. 

Question  3.3 

(i)  For  each  of  the  Markov  chains  in  Questions  3.1  and  3.2,  use  Theorem  3.1  to 
decide  whether  or  not  it  has  a  unique  stationary  distribution. 

(ii)  Find  a  stationary  distribution  for  each  of  these  Markov  chains  by  finding  a 

solution  of  Equations  (2.7)  and  (2.9)  which  satisfies  Condition  (2.8).  Is  your 
solution  unique?  Check  that  your  answers  here  agree  with  the  conclusions  you 
reached  in  part  (i)  by  using  Theorem  3.1.  □ 


3.2  Periodicity 

Theorem  3.1  gives  conditions  for  a  finite  Markov  chain  to  have  a  unique 
stationary  distribution;  and  from  Subsection  2.3,  if  a  Markov  chain  satisfying  these 
conditions  has  a  limiting  distribution,  then  the  limiting  distribution  is  the  unique 
stationary  distribution  of  the  chain.  However,  a  finite  Markov  chain  that  has  a 


Solution  2.7  and  Example  2.3. 


Examples  1.4,  1.5  and  1.6 
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unique  stationary  distribution  may  not  have  a  limiting  distribution  (see  Example 
2.4).  So  what  additional  conditions  are  necessary  to  guarantee  that  the  n-step 
transition  probabilities  converge  to  the  unique  stationary  distribution?  To  try  to 
answer  this,  we  shall  first  look  again  at  the  behaviour  of  the  Markov  chain  in 
Example  2.4. 


Example  2.4,  revisited 


The  Markov  chain  with  transition  matrix  P  = 


has  a  unique  stationary 


distribution  (by  Theorem  3.1).  But  in  Section  2  you  saw  that,  since  the  n-step 
transition  probabilities  oscillate  between  0  and  1  as  n  increases,  no  limiting 


distribution  exists.  For  m=  1,2,  ...,  we  have  P2m 
so  in  particular, 


1 

0 


0 

1 


and  P2m  +  1 


1 

0 


Poo  = 


if  n  is  even 
if  n  is  odd, 


( 1  if  n  is  even 
(0  if  n  is  odd. 


So  a  return  to  state  0,  or  a  return  to  state  1,  can  occur  only  after  an  even  number 
of  steps.  States  0  and  1  are  said  to  be  periodic  with  period  2.  □ 


In  general,  a  state  i  is  said  to  be  periodic  if  there  is  an  integer  k  greater  than  1  such 
that  the  times  at  which  a  return  to  state  i  is  possible  are  all  multiples  of  k;  the 
period  of  state  i  is  the  greatest  such  integer  k.  If  there  is  no  such  integer  k  greater 
than  1,  then  state  i  is  aperiodic. 


For  example,  in  an  unrestricted  random  walk  with  p  +  q  =  1,  returns  to  the  origin 
can  occur  only  at  even-numbered  times,  so  state  0  has  period  2.  On  the  other 
hand,  in  an  unrestricted  simple  random  walk  with  p  +  q  +  r  =  1,  where  r  >  0, 
returns  to  the  origin  can  occur  after  any  number  of  steps,  so  state  0  is  aperiodic. 


Example  3.3 


For  the  Markov  chain  with  transition  matrix 


P  = 


0 

1 

2 

3 


0  10  0 
0  0  10 
0  0  0  1 
10  0  0 


which  is  illustrated  in  Figure  3.3,  return  to  a  state  is  possible  only  after  4,  8,  12,  ... 
transitions;  these  times  are  all  multiples  of  4  and  the  period  of  each  state  is  4. 
Notice  that  although  these  times  are  also  all  multiples  of  2,  the  period  of  a  state  is 
defined  to  be  the  greatest  integer  of  which  the  possible  return  times  are  all 
multiples,  so  the  period  of  each  state  is  4  and  not  2.  □ 


Question  3.4 
P  = 


A 

0 

1 

2 


Markov  chain  has  transition  matrix 

‘i  i  0“ 

0  0  1. 

0  1  0 


For  each  state,  decide  whether  it  is  periodic  or  aperiodic;  if  it  is  periodic,  state  its 
period.  □ 


The  distribution  of  Z„,  the  nth  step 
in  a  simple  random  walk,  is: 
P(ZB=l)  =  p,  P(Zn=-l)  =  q, 
P(Z„  =  0)  =  r.  See  Unit  5, 
Subsection  1.1. 


Figure  3.3 


Several  more  examples  of  finding  the 
periods  of  the  states  of  a  Markov 
chain  are  given  in  the  audio-tape 
session  (Subsection  4.2). 


In  Example  3.3  all  the  states  have  period  4;  and  for  a  simple  random  walk  with 
P  +  q  =  1,  each  state  has  period  2.  In  Question  3.4,  however,  state  0  is  aperiodic 
and  states  1  and  2  have  period  2;  so  in  a  Markov  chain,  periodic  and  aperiodic 
states  can  arise.  In  fact,  not  all  periodic  states  need  have  the  same  period. 
However,  in  each  of  these  examples,  states  which  communicate  do  have  the  same 
period — the  Markov  chain  in  Example  3.3  and  a  simple  random  walk  with 
P  +  q  =  1  are  irreducible,  and  the  Markov  chain  in  Question  3.4  has  two 
communicating  classes,  {0}  and  {1,2}.  That  communicating  states  have  the  same 
period  is  not  a  property  specific  to  these  examples;  it  is  a  general  property  of 
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Markov  chains.  Any  two  states  which  communicate  either  have  the  same  period  or 
are  both  aperiodic.  That  is,  periodicity  is  a  class  property: 

either  every  state  in  a  communicating  class  is  aperiodic,  or  every  state 
(in  a  class)  is  periodic  with  period  k  for  some  k  >  2,  k  being  the  same 
for  every  state  in  the  class. 

This  means  that,  when  investigating  the  periodicity  of  the  states  of  a  Markov 
chain,  it  is  not  necessary  to  check  every  state;  it  is  sufficient  to  check  one  state  in 
each  class.  It  also  allows  us  to  talk  about  aperiodic  classes  and  periodic  classes,  and 
the  period  of  a  class  of  states;  so  in  Question  3.4,  the  communicating  class  {1,2}  is 
periodic  with  period  2.  And  if  a  Markov  chain  is  irreducible,  then  we  can  describe 
the  chain  as  a  whole  as  periodic  or  aperiodic.  Thus,  a  simple  random  walk  with 
p  +  q  =  1  is  periodic  with  period  2,  a  simple  random  walk  for  which  r  >  0  is 
aperiodic,  and  the  Markov  chain  in  Example  3.3  is  periodic  with  period  4. 

Periodicity  and  limiting  distributions 

As  you  have  just  seen,  the  (irreducible)  Markov  chain  in  Example  2.4  is  periodic 
with  period  2,  but  it  does  not  have  a  limiting  distribution.  On  the  other  hand,  the 
(irreducible)  Markov  chain  models  for  rainfall  patterns  and  job  mobility  are 
aperiodic,  and  for  each  the  limiting  distribution  exists  and  is  the  unique  stationary 
distribution  of  the  chain.  This  suggests  that  aperiodicity  of  an  irreducible  Markov 
chain  may  be  a  necessary  condition  for  the  existence  of  a  limiting  distribution. 

This  is  indeed  the  case;  and  it  can  be  shown  that  for  a  finite  irreducible  Markov 
chain,  aperiodicity  is  also  sufficient,  although  the  proof  that  this  is  so  is  not  given 
here.  This  result  is  contained  in  Theorem  3.2  below,  which  says  that  every  finite 
irreducible  aperiodic  Markov  chain  has  a  limiting  distribution  which  is  also  the 
unique  stationary  distribution  of  the  chain.  (The  stationary  distribution  is  unique 
by  Theorem  3.1.) 


Theorem  3.2 

If  an  irreducible  Markov  chain  with  finite  state  space  {0, 1,  ...,  m}  is 
aperiodic,  then  for  all  states  i  and  j, 

p\f  — >  7 ij  as  72  — >  co, 

where  n  =  [ tc0  nl  ...  7rm]  is  the  unique  stationary  distribution  of  the  chain 
(that  is,  the  limiting  distribution  is  the  stationary  distribution). 


In  Section  2,  when  discussing  the  long-run  behaviour  of  a  Markov  chain,  in  a 
number  of  examples  the  existence  of  a  limiting  distribution  was  assumed.  The 
importance  of  this  theorem  is  that  it  shows  that  this  assumption  was  valid;  the 
rainfall  models  (Examples  1.4  and  1.5),  the  job  mobility  model  of  Example  1.6  and 
Markov’s  model  based  on  Pushkin’s  ‘Eugene  Onegin’  (Example  2.2)  are  all  finite 
irreducible  aperiodic  Markov  chains.  Another  example  of  the  application  of  this 
theorem  can  be  found  in  Subsection  5.1  of  Unit  5;  there,  a  gambling  scenario  was 
discussed  and  it  was  stated  that  the  probability  settles  down  as  n  ->  co.  The 
process  in  this  model  is  a  finite  irreducible  aperiodic  Markov  chain,  so 
Theorem  3.2  is  the  justification  for  the  statement. 

This  simple  but  powerful  theorem  says  that  any  irreducible  finite  Markov  chain 
that  has  been  running  for  a  long  time  can  be  assumed  to  be  approximately  in  its 
unique  equilibrium — the  unique  stationary  distribution — provided  it  satisfies  the 
condition  of  aperiodicity. 
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4  The  classification  of  states 


In  Subsection  4.1,  the  ideas  of  recurrence  and  transience,  which  were  introduced 
for  random  walks  in  Unit  5,  are  extended  to  Markov  chains.  Subsection  4.2 
contains  an  audio-tape  session  which  consists  of  examples  illustrating  the  ideas 
and  results  of  Sections  3  and  4;  no  new  ideas  are  introduced  in  this  subsection. 


4.1  Recurrent  and  transient  states 


The  idea  of  the  recurrence  or  transience  of  a  random  walk  was  introduced  in 
Unit  5;  an  unrestricted  random  walk  is  recurrent  if  a  return  to  the  origin  is  certain 
to  occur  sooner  or  later,  and  transient  if  eventual  return  is  not  certain.  For  a 
simple  random  walk  with  p  +  q  =  1,  it  was  shown  that  if  p  =  q  =  i,  then  the 
probability  that  a  return  to  the  origin  will  occur  eventually  is  1,  but  that  this 

probability  is  less  than  1  if  p  #  q;  so  such  a  simple  random  walk  is  recurrent  if  See  Unit  5,  Subsection  3.1. 

P  =  q  =  i  and  transient  if  p  #  q.  Although  we  considered  the  probability  of  return 

to  the  origin,  we  could  equally  well  have  found  the  probability  of  return  to  any 

other  state  the  result  would  have  been  the  same.  For  a  random  walk,  the  steps 

(or  increments)  are  independent  of  the  current  position;  so,  wherever  the  particle 

starts  from,  its  probability  of  returning  there  is  the  same.  However,  this  is  not  the 

case  for  Markov  chains  in  general;  Example  4.1  illustrates  this  point. 


Example  4.1 


For  the  two-state  Markov  chain  with  transition  matrix 


0 

1 


,  which  is 


illustrated  in  Figure  4.1,  starting  from  state  0  the  probability  of  a  return  to  state  0 
is  1  (since  p00  =  1);  but  from  state  1,  the  probability  of  leaving  immediately  (i.e.  at 
the  next  transition)  and  never  returning  is  {  (since  pl0  =  \  and  p00  =  1),  so  a 
return  to  state  1  is  not  certain  to  occur.  So  for  one  state  a  return  is  certain  to 
occur  and  for  the  other  it  is  not;  and  we  cannot  speak  of  the  Markov  chain  as  a 
whole  as  recurrent  or  transient.  □ 


i 

1  T 


0  - < -  1 

i 


Figure  4.1 


However,  for  Markov  chains  in  general,  we  can  speak  of  the  individual  states  as 
being  recurrent  or  transient:  state  i  is  recurrent  if  a  return  to  state  i  is  certain  to 
occur  eventually;  state  i  is  transient  if  a  return  to  state  i  is  not  certain— that  is, 

state  i  is  transient  if  there  is  a  positive  probability  of  leaving  (immediately)  and  The  word  ‘immediately’  is  normally 

never  returning.  So,  in  Example  4.1,  state  0  is  recurrent  and  state  1  is  transient.  omitted  in  such  descriptions. 


Example  4.2  The  gambler’s  ruin 

The  gambler’s  ruin  Markov  chain  is  illustrated  in  Figure  4.2;  this  process  is  a 
random  walk  with  two  absorbing  barriers. 


Figure  4.2  Transitions  for  the  gambler’s  ruin 


By  definition,  once  an  absorbing  state  is  entered,  the  process  is  certain  to  remain 
in  that  state  thereafter — it  returns  after  every  step — so  an  absorbing  state  is 
recurrent.  In  gambling  terminology,  once  a  player  is  ruined  he  can  never  win  any 
money  back— he  is  permanently  ruined;  so  states  0  and  a  are  recurrent.  But  what 
about  states  1,2,  ...,  a  —  1?  Since  eventual  ruin  of  one  or  other  of  the  gamblers  is 
certain,  surely  these  states  are  transient?  To  show  that  a  state  is  transient  we  must 
show  that  there  is  a  positive  probability  of  leaving  and  never  returning.  Suppose 
that  Gary  has  £j  (0  <j<  a) — that  is,  the  Markov  chain  is  in  state  j.  The 
probability  that  he  is  ruined  in  j  bets — the  shortest  possible  time — is  qj;  so  the 
probability  of  never  again  having  £j  is  at  least  qj,  which  is  greater  than  zero.  That 
is,  the  probability  of  never  returning  to  state  j  is  positive,  and  so  this  state  is 
transient.  Hence  states  1,2,  ...,  a  —  1  are  all  transient,  whatever  the  value  of  p.  □ 
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Question  4.1  For  the  Markov  chain  in  Example  1.2  (Collecting  animals),  which 
states  are  recurrent  and  which  are  transient?  □ 

The  recurrence  and  transience  of  the  states  of  the  Markov  chains  considered  so  far 
in  this  subsection  can  be  summarized  in  the  following  way.  A  simple  random  walk 
with  p  +  q  —  1  is  an  irreducible  Markov  chain  and  either  all  the  states  are 
recurrent  (p  =  q  =  i)  or  all  are  transient  (p  ^  <?)•  For  the  gambler’s  ruin 
(Example  4.2),  there  are  three  communicating  classes:  {0}  and  {a},  which  contain 
recurrent  states,  and  {1,2,  ...,  a  —  1}  which  contains  only  transient  states.  In 
Example  4.1,  there  are  two  classes,  {0}  and  {1} — state  0  is  recurrent  and  state  1  is 
transient — and  in  Question  4.1,  the  state  in  class  {3}  is  recurrent  and  the  states  in 
the  classes  {0},  {1}  and  {2}  are  transient. 

For  each  of  these  Markov  chains,  either  all  the  states  in  a  class  are  recurrent  or  all 
are  transient.  In  Section  5  you  will  see  that  this  is  true  for  every  Markov  chain; 
that  is,  either  every  state  in  a  communicating  class  is  recurrent  or  every  state  is 
transient — recurrence  is  a  class  property.  This  is  why,  in  Unit  5,  we  were  able  to 
refer  to  a  random  walk  as  a  whole  as  recurrent  or  transient  instead  of  referring  to 
individual  states  as  recurrent  or  transient.  And  when  investigating  the  recurrence 
or  transience  of  the  states  of  a  Markov  chain  we  do  not  have  to  check  each  state 
individually;  it  is  necessary  to  check  only  one  state  in  each  class.  (In  Unit  5 ,  for  a 
simple  random  walk,  we  checked  the  origin.)  Since  recurrence  is  a  class  property, 
we  can  describe  a  class  as  recurrent  or  transient;  for  example,  in  the  gambler’s 
ruin,  {0}  and  { a }  are  recurrent  classes  and  (1,2,  ...,  a  —  1}  is  a  transient  class. 

And  if  a  Markov  chain  is  irreducible,  then  we  can  describe  the  chain  as  a  whole  as 
recurrent  or  transient. 

This  subsection  closes  by  giving  (without  proof)  a  few  useful  recurrence  properties 
of  Markov  chains;  in  many  examples,  these  allow  us  to  decide,  without  doing  any 
calculations,  whether  states  are  recurrent  or  transient. 

It  is  often  possible  to  determine  whether  a  class  is  recurrent  or  transient  simply  by 
looking  to  see  if  it  is  closed.  Every  class  which  is  not  closed  is  transient,  whether  it 
has  a  finite  or  an  infinite  number  of  states;  this  is  so  since  there  is  a  positive 
probability  of  leaving  a  class  which  is  not  closed  and  then  return  is  impossible.  On 
the  other  hand,  every  finite  closed  class  is  recurrent;  so,  in  particular,  every  finite 
irreducible  Markov  chain  is  recurrent.  Thus,  for  example,  the  Markov  chains  used 
to  model  rainfall  patterns  and  job  mobility  are  all  recurrent. 

Example  4.3 

For  the  Markov  chain  in  Example  3.1,  which  is  illustrated  in  Figure  4.3,  {0, 1}  and 
{3}  are  closed  classes,  but  {2}  is  not  closed,  so  states  0,  1,  3  are  recurrent  and  state 
2  is  transient.  □ 


Figure  4.3 


Question  4.2  For  each  of  the  Markov  chains  in  Question  3.1,  determine  which 
states  are  recurrent  and  which  are  transient.  □ 

The  only  type  of  class  not  covered  by  the  two  rules  given  above  is  an  infinite 
closed  class,  which,  in  fact,  can  be  either  transient  or  recurrent.  You  have  already 
met  an  example  which  shows  that  this  is  so:  the  states  of  a  simple  random  walk 
with  p  +  q  =  1  form  a  single  closed  class,  and  the  random  walk  is  recurrent  if 
p  =  \  and  transient  if  p  # 


Page  4 


This  is  an  obvious  property  of 
single-state  classes,  which  you  met  in 
Example  4.1  and  Question  4.1. 


Page  23 
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The  recurrence  and  transience  of  Markov  chains  will  be  discussed  further  in 
Section  5;  the  properties  given  in  this  subsection  are  summarized  in  Table  4.1 


Table  4.1  Recurrence  properties  of  communicating  classes 


Type  of  class 

Finite 

Infinite 

Closed 

Not  closed 

Recurrent 

Transient 

Recurrent  or  transient 
Transient 

4.2  Classifying  the  states  of  a  finite  Markov  chain 


In  this  audio-tape  session,  the  main  results  of  this  section  and  the  previous  one 
will  be  summarized  as  they  apply  to  Markov  chains  with  finite  state  spaces.  We 
shall  work  through  a  number  of  examples;  in  each  we  shall  identify  the 
communicating  classes  of  the  Markov  chain,  decide  whether  or  not  each  class  is 
closed,  whether  the  states  are  recurrent  or  transient,  and  whether  they  are  periodic 
or  aperiodic.  We  shall  also  apply  Theorems  3.1  and  3.2  to  decide  which  chains 
have  a  unique  stationary  distribution  and  which  have  a  limiting  distribution  You 
will  need  paper  to  write  on  as  you  work  through  the  examples 


Now  switch  on  the  tape. 
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VT)  Classifying  the  states  of  a  finite  Markov  chain 

Method 

(1)  Draw  a  diagram  for  the  Markov  chain. 

(2)  Identify  the  communicating  classes. 

(3)  Decide  whether  or  not  each  class  is  closed  and  hence  whether  it  is  recurrent 
or  transient. 

(4)  Decide  whether  each  class  is  periodic  or  aperiodic ;  if  a  class  is  periodic, 
find  its  period. 

V _ J 


\ 

Example 

Classify  the  states  of  the  Markov  chain  which  has.  the  following  transition  matrix  P. 


0 


P  = 


2 

3 


0.2 

0 

0 

0.1 


0.4  0  0.4 

0.5  0.5  0 

1  0  0 

0  0  0.9 


Solution 


(1)  A  diagram  for  this  Markov  chain  is  shown. 

Q 

O 

i 

_0_ 

— > — 

1 

\ 

t 

3 

1  1 

2 

0 

(2)  The  commiArwcatmq  classes  are  j  0, 3  j  and  { 1 ,  2  } . 

(3)  [l,2j  is  closed  -  there  are  no  arrows  leaving  if  -  and  hence  is  recurrent. 

[0,3}  is  not  closed,  so  if  is  transient. 

(4)  The  classes  f 0, 3  J  and  { 1 , 2  ]  are  both  aperiodic. 

V _ _ _ 
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Question 

Classify  the  states  of  the  Markov  chains  having  the  following 
transition  matrices. 


0  0  I 


1  i 

2  2 


1  0  0 

I  0  0 


0  0  0 


0  0 


0  0 


0.8  0 

0  0 

0  0 

0.9  0 

0  0 

0.3  0.5 


0.2  0  0  0 

0.1  0  0.9  0 

0  10  0 
0.1  0  0  0 

0  0  0  1 

0  0  O.Z  0 


0  0  10  0  0  0 


Problem 

Does  the  Markov  chain  in  Frame  Z  have  a  unique  stationary  distribution  ? 


Solution 

This  Markov  chain  has  only  one  closed  class,  j  1,  2  J ,  so,  by  Theorem  3. 1, 
it  has  a  unique  stationary  distribution. 


Question 

For  each  of  the  Markov  chains  in  Frame  3,  decide  whether  or  not  it  has 
a  unique  stationary  distribution.  And,  for  each  irreducible 
Markov  chain,  apply  Theorem  3.2  to  decide  whether  or  not  it  has  a 
limiting  distribution. 
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Communicating  classes:  ( 0, 1 ),  (cj. 

{ 0, 1  j  is  not  closed ,  transient,  aperiodic. 

{2  J  is  closed ,  recurrent,  aperiodic. 

{0, 1,2]  is  irreducible,  recurrent, 
periodic  with  period  2. 

{0,1,2}  is  irreducible,  recurrent, 
aperiodic. 

Communicating  classes  :  f 0, 4} ,  {2,3},  {l }. 
{0, 4}  is  closed,  recurrent. 

{l }  and  {2,3}  are  not  closed,  transient. 

{O,  4}  is  periodic  with  period  2. 
jl  J  and  {2,  3}  are  aperiodic. 

Communicating  classes:  {2,4,6},  [ 0, 3 ] ,  { 1, 5 } 
{2,4,6}  is  closed,  recurrent,  periodic 
with  period  3. 

{0, 3]  is  closed,  recurrent,  aperiodic. 

{1,5}  is  not  closed,  transient,  aperiodic. 


Of  the  Markov  chains  in  Frame  3,  only  (V)  has  more  than  one  closed  class, 
so  all  except  (v)  have  a  unique  stationary  distribution. 


The  Markov  chains  (ii)  and  (Hi)  are  irreducible;  (iii)  is  aperiodic  and 
(11)  is  periodic,  so  (iii)  has  a  limiting  distribution  and  (ii)  does  not. 


5  Return  probabilities  and  return  times 


In  this  section  the  recurrence  and  transience  of  the  states  of  a  Markov  chain  will 
be  discussed  in  more  detail.  In  Unit  5  the  return  probabilities  and  first  return 
probabilities  were  defined  for  a  random  walk.  In  Subsection  5.1, 
the  corresponding  probabilities  will  be  defined  for  a  state  i  of  a  Markov  chain; 
and  a  criterion  for  the  recurrence  of  a  state  i  will  be  derived,  similar  to  that 
obtained  in  Unit  5  for  a  random  walk.  This  criterion  and  the  Chapman- 
Kolmogorov  equations  will  be  used  to  prove  that  recurrence  is  a  class  property. 
Finally,  in  Subsection  5.2  for  a  simple  example,  we  shall  find  the  expected  time  of 
first  return  to  a  recurrent  state;  and  a  theorem  is  stated  which  provides  a  method 
of  calculating  the  mean  return  time  of  any  state  in  an  aperiodic  irreducible 
recurrent  Markov  chain. 


5.1  Transience  and  recurrence 


In  Section  1,  for  n  >  0  and  states  i  and  j,  the  n-step  transition  probability,  pW,  was 
defined  to  be  the  probability  that,  starting  from  state  i,  the  Markov  chain  is  in 
state  j  after  n  transitions:  pg>  =  P(Xn  =j\X0  =  i).  In  particular,  for  n  >  1,  pW  is 
the  probability  that,  starting  from  state  i,  the  Markov  chain  is  again  in  state  i,  not 
necessarily  for  the  first  time .  after  n  transitions;  of  course,  p|f)  =  1.  The 
probabilities  p§\  n  =  0,1,...,  are  the  return  probabilities  for  state  i.  For  n  >  1,  the 
first  return  probability  for  state  /,  f^\  is  the  probability  that,  starting  from  state  i, 
the  Markov  chain  is  again  in  state  i  for  the  first  time  after  n  transitions; /|0)  is 
defined  to  be  0.  Adding  these  first  return  probabilities  gives  the  probability  that  a 
return  will  occur  at  some  time,  sooner  or  later;  the  probability  that  a  return  to 
state  i  will  occur  eventually  is  denoted  by  fih  so  we  have 

State  i  is  recurrent  if  fu  =  1  and  transient  if  /„  <  1. 

In  the  next  example,  this  criterion  is  applied  to  one  state. 


Example  5.1  Melbourne  weather 

The  Markov  chain  used  in  Example  1.4  to  model  the  weather  in  Melbourne  has 
two  states,  0  (dry)  and  1  (wet),  and  transition  matrix  P  given  by 

1  -  a  a 

P  l-PJ 

where  a  =  0.174,  p  =  0.600.  The  first  return  to  state  0  occurs  after  one  transition  if 
the  first  transition  is  from  0  to  0,  so 

/oo)  =  1  —  a. 

The  first  return  to  state  0  occurs  after  two  transitions  if  the  first  transition  is  from 
0  to  1  and  the  second  is  from  1  to  0,  so 

foo  =  «/?• 

For  n  >  3,  the  first  return  to  state  0  occurs  after  n  transitions  if  the  first  transition 
is  from  0  to  1,  the  next  n  —  2  transitions  are  from  1  to  1  and  the  nth  is  from  1  to 
0;  so 

fSS  =  «(1  -  fl“2p. 

In  fact,  this  formula  is  also  valid  for  n  =  2. 


The  probability  that  a  return  to  state  0  occurs  eventually  is  the  sum  of  these  first 
return  probabilities;  that  is, 


/oo  =  1  -  a  +  £  a(l  —  P)n~2(3 


1  —  a  +  a/?(l  +  (1  —  /?)  +  (1  —  /?)2  +  . . .) 
ocfi 


=  I-“  +  i-d-/o 

So  state  0  is  recurrent.  □ 


=  1  -  a  +  a  =  1. 


This  argument  is  valid  for  0  <  a  <  1, 

0<p<  1. 
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Question  5.1  The  diagram  in  Figure  5.1  represents  the  Markov  chain  model  for 
success  runs  in  a  Bernoulli  process  which  was  described  in  Example  1.3. 


— 

— 

a 

0 

2 

_ 

3 

\ 

- > - 

P 

1 

- > - 

p 

P 

/ 

P 

Figure  5.1  Transition  probabilities  for  a  success-runs  Markov  chain 


(i)  Find  the  first  return  probabilities  for  n  >  1. 

(ii)  Show  that  state  0  is  recurrent.  □ 

For  an  unrestricted  random  walk,  the  return  probability  p&>  has  the  same  value  for 
every  state  i;  in  Unit  5,  Section  4,  this  common  probability  was  denoted  by  un. 
Similarly,  the  first  return  probability  has  the  same  value  for  every  state  i;  this 
was  denoted  by  f„.  And  fh  the  probability  that  a  return  to  state  i  will  occur 
eventually,  was  denoted  by  /.  For  n  >  1,  you  established  the  identity 

Ml,  =foUn  +  flUn-l  +  fzUn-2  +  •••  +  /n-  1W1  +  /"U0> 

and  for  the  generating  functions  F(s)  and  U(s)  of  the  probabilities  {/„}  and  {«„}  it 
was  shown  that 

U{s)  =  1  +  F{s)  U(s). 

This  was  used  to  show  that  a  random  walk  is  recurrent  if  and  only  if  the  sum  of 
the  return  probabilities,  u„,  is  infinite;  that  is ,/=  1  if  and  only  if  =  oo.  (A 
rigorous  proof  of  this  result  was  not  given.) 

In  the  next  two  questions  you  are  asked  to  obtain  parallel  results  for  a  state  i  of  a 
Markov  chain.  The  method  used  is  identical  to  that  used  in  Unit  5,  so  you  will 
find  it  helpful  to  have  that  unit  open  at  Subsection  4.1  as  you  work  through  these 
questions.  If  you  are  short  of  time,  you  may  prefer  to  leave  these  questions  until 
you  revise  Units  5  and  6. 


Question  5.2 

(i)  Show  that 

Pu'  +/ii2,p!?) 

and 


p\f'  +/,^1)P^,?,  +/u2H1) 

(ii)  Find  a  similar  expression  for  p)?’.  □ 


Corresponding  to  U{s)  and  F(s)  for  a  random  walk,  are  two  generating  functions, 
P^s)  and  Fu(s),  associated  with  state  i  of  a  Markov  chain: 

OO 

Pu(s)  =  I  p!fs",  |S|  <  1. 

n  =  0 

F„(s)  =  t  fu's"-  lsl  <  1- 

n  =  0 

(Recall  that  =  1  and  =  0.) 


Question  5.3 

(i)  Use  the  identity  that  you  found  in  Solution  5.2(ii)  to  show  that,  for  ieS, 

Pu(s)  =  1  +Fu(s)Pu(s).  (5.1) 


(ii) 


Use  this  result  to  show  that 

state  i  is  recurrent  if  and  only  if  the  sum  of  the  return  probabilities, 

CO 

pjf,  is  infinite;  that  is,  fu  =  1  if  and  only  if  £  p|f  =  oo. 

n  =  0 

(A  rigorous  proof  is  not  required.)  □ 


Unit  5,  Question  4.1 

These  two  results  are  Identities  (4.2) 
and  (4.3)  of  Unit  5,  Subsection  4.1. 
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We  now  have  all  the  results  needed  in  order  to  prove,  as  promised  in  Section  4, 
that  recurrence  is  a  class  property;  that  is,  two  states  in  the  same  communicating 
class  are  either  both  recurrent  or  both  transient.  This  will  be  done  by  showing  that 
if  i  and  j  are  communicating  states  and  i  is  recurrent,  then  j  is  also  recurrent.  This 
proof  is  quite  difficult,  so  do  not  worry  if  you  cannot  follow  all  the  details;  you 
will  not  be  expected  to  reproduce  the  proof. 

If  states  i  and  j  communicate,  then  there  exist  integers  r,  s  >  1  such  that 

Pij  >  0  and  Pjf  >  0.  For  any  integer  m  >  0,  applying  the  Chapman-Kolmogorov 

equations  (1.8),  we  have 

PX+’+”’  =  J  pgp&+»>  >  pWpg+»>.  (5.2) 

keS 

Applying  the  Chapman-Kolmogorov  equations  a  second  time,  we  have 

„(r  +  m)  _  Y  _(m)_(r)  (m)  (r)  .  .. 

Pij  ~  L  Pik  Pkj  ^  Pii  Pij  •  (5.3) 

keS 

So,  combining  Inequalities  (5.2)  and  (5.3),  we  have 

n(r  +  s+m)  >  (s)n(m)  (r)  . 

Pjj  ^  Pji  Pii  Pij  •  (5.4) 

But 

E  Pj}  *  E  Pi} 

n  =  0  n  =  r  +  s 

=  E  p%*‘*m) 

m  =  0 

^  Z  pfpft'py,  by  Inequality  (5.4), 

m  =  0 

=  PfM}  E  Pu'-  (5.5) 

m  =  0 

oo 

Since  state  i  is  recurrent,  it  follows  from  Question  5.3(ii)  that  J  =  oo.  Hence, 

m  =  0 

from  Inequality  (5.5)  Zp5?  infinite,  so  state  j  is  also  recurrent.  It  follows  that  it 
is  not  possible  for  only  one  of  a  pair  of  communicating  states  to  be  recurrent;  they 
must  be  either  both  recurrent  or  both  transient.  Hence,  either  all  the  states  in  a 
communicating  class  are  recurrent  or  all  are  transient. 


5.2  Return  times  for  recurrent  states 

A  state  i  of  a  Markov  chain  is  recurrent  if  fu,  the  sum  of  the  first  return 
probabilities,  is  equal  to  1 ;  that  is,  if  the  Markov  chain,  starting  from  i,  is  certain 
sooner  or  later  to  return  to  state  i.  But  if  a  return  is  certain  to  occur,  when  will  it 
take  place?  Is  it  likely  to  be  after  only  a  few  transitions  or  is  it  more  likely  that  a 
long  period  will  elapse  before  state  i  is  revisited?  And  what  is  the  average  number 
of  transitions  that  will  take  place  before  state  i  is  revisited? 

The  first  return  probability  fW  is  the  probability  that  the  first  return  to  state  i 
occurs  after  n  transitions;  and  for  a  recurrent  state,  the  sum  of  the  probabilities 
f("\  n=  1,2,3, is  1.  So,  for  a  recurrent  state  i,  {f^:  n  >  1}  is  the  distribution 
of  the  time,  Th  at  which  the  first  return  to  state  i  occurs.  The  mean  of  this 
distribution,  which  is  denoted  by  ph  is  called  the  mean  (first)  return  time  or  the 
mean  recurrence  time  of  state  i. 


Example  5.2  Unrestricted  simple  random  walks 

In  Unit  5  (Subsections  3.1  and  4.2)  you  saw  that  a  simple  random  walk  with 

P  =  Q  =  7  is  recurrent.  The  first  return  probabilities,  /„,  for  a  simple  random  walk  fja)  =/„  for  all  i. 
with  p  +  q  =  1  were  obtained  in  Subsection  4.2  of  that  unit;  and  in  finding  them  it 
was  shown  that  F(s),  the  generating  function  of  these  probabilities,  is  given  by 

F{s)  =  1  -  (1  -  4 pqs2)112.  This  is  Equation  (4.7)  of  Unit  5. 

When  p  =  q  =  \  this  reduces  to 

F(s)=  1  -(1  -s2)112. 

Since  the  sum  of  the  first  return  probabilities  is  1  in  this  case,  the  mean  of  the 

distribution  of  the  first  return  time  is  given  by  p  =  F'{  1).  Differentiating  F(s),  we  p,  =  p  for  all  i. 

have 

F\s)  =  s(  1  -s2)“1/2, 

so  F'(  1)  =  oo;  that  is,  the  mean  return  time  to  the  origin  (or  to  any  other  state)  is 
infinite  for  a  simple  random  walk  with  p  =  q  =  So  although  a  return  is  certain 
to  occur  eventually,  it  may  not  occur  until  a  very  long  time  has  elapsed.  □ 

Example  5.1,  continued 


For  the  Markov  chain  used  to  model  Melbourne’s  weather,  the  first  return 
probabilities  for  state  0  are  given  by 

foo  =  1  -  a, 

/oo  =  «(1  —  PY~2P,  n>  2. 

This  Markov  chain  is  recurrent  (because  it  is  finite  and  irreducible)  and  the  mean 
return  time  to  state  0  is  given  by  p0  =  Fq0(1),  where  F00(s)  is  the  p.g.f.  of  T0,  the 
time  of  the  first  return  to  state  0.  The  p.g.f.  is  given  by 

Foo(s)  =  t  ftS? 

n  =  0 

=  (1  -  a )s+  §  a(l  -P)n~2Psn 

n  =  2 


So 


and 


=  (1  -  a)s  +  otps2(  1  +  (1  -  p)s  +  (1  -  p)2s2  +  ...) 
aBs2 

=  (1  -  a)s  +  j  _  (1  _  since  |(1  —  P)s\  <  1. 


Foo(s)  —  1  —  ct  + 


2otPs(l  -  (1  -  P)s)  +  aps2(  1  -  P) 

(1-(1-P)s)2 


foo  =  0— see  page  33. 


Mo  —  ^oo(l)  —  1  —  a  + 


2 ap  x  P  +  a/?(l  -  p) 

P2 


which  simplifies  to  give 


Mo  = 


a+| 


So  for  the  Melbourne  weather  Markov  chain,  the  mean  time  between  dry  days  is 
(0.174  +  0.600)/0.600,  or  approximately  1.29  days.  □ 


Question  5.4  Show  that  the  mean  time  between  wet  days  for  the  Melbourne 
weather  Markov  chain  is  (a  +  p)/ot  ~  4.45  days.  □ 

In  Example  5.2,  the  expected  time  to  the  first  return  to  the  origin  is  infinite, 
whereas  in  Example  5.1  the  states  have  finite  mean  return  times.  If  the  mean 
return  time  to  a  recurrent  state  i  is  infinite,  then  state  i  is  said  to  be  null  recurrent; 
and  if  the  mean  return  time  to  state  i  is  finite,  then  state  i  is  said  to  be  positive 
recurrent.  So  the  origin  (and  hence  all  the  other  states)  of  the  simple  random  walk 
with  p  =  q  =  \  is  null  recurrent;  and  states  0  and  1  of  the  Markov  chain  for 
Melbourne’s  weather  are  positive  recurrent. 
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In  Subsection  5.1  you  saw  that  recurrence  is  a  class  property;  it  is  also  the  case 
that  positive  or  null  recurrence  is  a  property  shared  by  all  the  states  in  a 
communicating  class,  although  no  proof  is  given  that  this  is  so.  Thus,  we  can  refer 
to  positive  recurrent  and  null  recurrent  classes,  or,  in  the  case  of  irreducible  chains, 
to  positive  recurrent  and  null  recurrent  chains.  So  the  simple  random  walk  with 
p  =  q  =  j  is  null  recurrent  and  the  two-state  Markov  chain  used  to  model  the 
weather  in  Melbourne  is  positive  recurrent.  In  fact,  every  finite  closed  class  is 
positive  recurrent;  so,  in  particular,  every  finite  irreducible  Markov  chain  is 
positive  recurrent.  And  an  infinite  closed  class  can  be  transient,  null  recurrent  or 
positive  recurrent.  You  have  already  come  across  examples  of  transient  and  null 
recurrent  infinite  Markov  chains  in  unrestricted  simple  random  walks;  an  example 
of  an  infinite  Markov  chain  which  is  positive  recurrent  will  be  given  at  the  end  of 
this  section.  Table  5.1  below  has  been  obtained  by  modifying  Table  4.1  to  include 
positive  and  null  recurrence. 


Table  5.1  Recurrence  properties  of  communicating  classes 


Type  of  class 

Finite 

Infinite 

Closed 

Positive  recurrent 

Positive  recurrent 
or  null  recurrent 

Not  closed 

Transient 

or  transient 
Transient 

Even  in  the  two  simple' examples  discussed,  finding  the  mean  return  times  involved 
a  fair  amount  of  work.  It  is  easy  to  see  that,  for  more  complicated  Markov  chains, 
finding  them  by  first  finding  the  first  return  probabilities  could  be  a  quite 
formidable  task.  However,  there  is  an  alternative  method;  it  is  a  consequence  of 
the  following  important  theorem,  the  proof  of  which  is  beyond  the  scope  of  this 
course. 


Theorem  5.1  The  Basic  Limit  Theorem  for  Markov  Chains 

For  each  state  i  of  a  recurrent  irreducible  aperiodic  Markov  chain 
lim  p$  = 


1 


I  nff" 

n  =  0 


Pi 


From  Theorem  3.1,  every  finite  irreducible  Markov  chain  has  a  unique  stationary 
distribution  n  =  [n0  nx  ...  7tm];  and  from  Theorem  3.2,  if  that  chain  is  also 
aperiodic,  then,  for  each  state  i,  lim  /?■"'  =  7tf .  So  the  Basic  Limit  Theorem  says 

that  for  a  finite  irreducible  aperiodic  Markov  chain,  =  1/p,.  for  every  state  i. 
Hence,  the  mean  return  times  for  such  chains  can  be  found  by  first  finding  the 
unique  stationary  distribution  n. 


Example  5.3 

For  the  two-state  Markov  chain  with  transition  matrix 
1  —  a  a 

P  i -PS 

where  a  and  /?  are  not  both  zero  and  not  both  one,  the  stationary  distribution  is 
given  by 

n  =  Oo  1 

+  P  a  +  Pj 

This  was  obtained  in  Solution  2.6.  Using  this  result  and  the  Basic  Limit  Theorem, 
we  have: 

a  +  (i  a  +  B 

Po  =  o — »  Pi= - L. 

P  a 

These  are  the  expressions  obtained  in  Example  5.1,  continued,  and 
Question  5.4.  □ 


The  Basic  Limit  Theorem  is  not 
applicable  to  the  cases 
a  =  0,  0  <  p  <  1, 
and 

/?  =  0,  0  <  a  <  1, 
both  of  which  are  not  irreducible. 
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In  general,  it  is  easier  to  find  the  stationary  distribution  of  a  Markov  chain  than  it 
is  to  calculate  the  mean  return  times  by  finding  the  first  return  probabilities;  so  the 
simple  relationship  7r;  =  for  each  ieS,  provides  a  relatively  straightforward 
method  of  finding  the  mean  return  times  for  a  Markov  chain  satisfying  the 
hypotheses  of  Theorem  5.1.  The  chains  in  Question  5.5  are  of  this  type,  and  their 
stationary  distributions  have  been  found. 


Page  14 
Page  19 


Pages  12  and  20 

Question  5.6  Markov  chain  models  have  been  used  to  study  the  pattern  of 
diseased  and  healthy  trees  in  a  forest.  In  one  such  model  it  is  assumed  that  the 
forest  can  be  subdivided  into  areas  of  two  types:  gaps,  which  contain  only  healthy 
trees,  and  patches,  which  contain  both  diseased  and  healthy  trees.  So  each  tree  can 
be  classified  as  a  diseased  tree  (state  0),  as  a  healthy  patch  tree  (state  1)  or  as  a 
healthy  gap  tree  (state  2).  Each  tree  along  a  randomly-chosen  path  through  the 
forest  is  of  one  of  these  three  types.  Suppose  that  the  types  of  trees  encountered  as 
the  path  is  followed  can  be  modelled  by  a  Markov  chain  with  transition  matrix  P 
given  by 

0  r 0.3  0.1  0.6" 

P  =  1  0.1  0.3  0.6  . 

2  [o.i  ot  o.8_ 

(i)  Approximately  what  proportion  of  the  trees  encountered  on  a  long  path  are 
healthy? 

(ii)  On  average,  how  many  healthy  trees  are  there  along  the  path  between  one 
diseased  tree  and  the  next?  □ 

The  method  of  finding  the  mean  return  time  to  a  recurrent  state,  based  on 
Theorem  5.1,  may  at  first  appear  to  be  limited  in  its  usefulness,  for  as  stated  it 
applies  only  to  finite  irreducible  aperiodic  Markov  chains.  However,  it  can  be 
generalized  in  two  ways.  Firstly,  any  finite  closed  class  can  be  treated  as  a  finite 
irreducible  Markov  chain  for  the  purpose  of  finding  the  mean  return  times  of  the 
states  in  the  class;  this  is  so  since,  by  definition,  once  the  Markov  chain  enters  a 
closed  class  it  can  never  again  leave  it;  and  hence  the  Markov  chain  behaves 
thereafter  as  if  its  only  states  were  those  in  the  closed  class.  Secondly,  the  Basic 
Limit  Theorem  (Theorem  5.1)  applies  to  any  recurrent  irreducible  aperiodic 
Markov  chain,  whether  finite  or  infinite.  So,  if  such  a  Markov  chain  with  an 
infinite  state  space  has  a  stationary  distribution  n,  then  7 r,-  =  l//r,.  for  each  state  i; 
and  in  this  case,  the  chain  is  positive  recurrent.  And  if  no  stationary  distribution 
exists  then  the  chain  is  null  recurrent.  So,  in  practice,  this  method  can  be  used  to 
find  the  mean  return  times  of  the  states  of  any  irreducible  recurrent  aperiodic 
Markov  chain  or  any  closed  recurrent  aperiodic  class  of  states. 


Question  5.5 

(i)  Find  the  average  number  of  days  between  purchases  of  brand  A  baked  beans 
by  the  student  in  Example  2.1. 

(ii)  Find  the  average  number  of  letters  from  one  consonant  to  the  next  in  the 
Markov  chain  model  in  Example  2.2  based  on  Pushkin’s  ‘Eugene  Onegin’. 

(iii)  Find  the  mean  number  of  generations  within  a  family  between  male  wage- 

earners  in  the  upper  occupational  level,  for  the  job  mobility  model  of 
Examples  1.6  and  2.3.  □ 
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Example  5.4  Success  runs  in  Bernoulli  trials 

In  Example  1.3  a  Markov  chain  model  for  success  runs  in  Bernoulli  trials  was 
described;  the  first  return  probabilities  were  found  in  Solution  5.1,  and  it  was 
shown  that  state  0  is  recurrent.  Since  recurrence  is  a  class  property  and  the 
Markov  chain  is  irreducible,  all  the  states  are  recurrent.  But  is  the  chain  positive 
recurrent  or  is  it  null  recurrent? 

The  distribution  of  T0,  the  time  at  which  the  first  failure  occurs,  is  given  by 
/do  =  T(T0  =  n)  =  pn~1q,  n>  1. 

So  T0  has  a  geometric  distribution,  T0  ~  G ^p),  and  the  mean  time  between  failures 
is  Ho  =  E(T0 )  =  \/q.  This  is  finite,  so  state  0,  and  hence  every  other  state,  is  positive 
recurrent. 

What  is  the  mean  number  of  trials  between  runs  of  at  least  k  successes?  This  is  the 
mean  time  between  visits  to  state  k;  that  is,  nk,  the  mean  return  time  for  state  k. 
This  can  be  found  by  calculating  n,  the  stationary  distribution  of  the  Markov 
chain,  and  then  using  nk  =  \/nk.  You  are  asked  to  find  this  stationary  distribution 
and  the  mean  return  times  in  the  final  question  of  this  unit. 

Question  5.7 

(i)  Find  the  stationary  distribution  of  the  Markov  chain  for  success  runs  in 
Bernoulli  trials. 

(ii)  Find  the  mean  number  of  trials  between  success  runs  of  length  at  least  k.  □ 


Objectives 


After  studying  this  unit  you  should  be  able  to: 

find  the  transition  matrix  of  a  Markov  chain  in  any  given  example; 

derive  the  Chapman-Kolmogorov  equations  and  apply  them  to  show  that  the 
n-step  transition  probabilities  are  the  entries  in  the  nth  power  of  the  transition 
matrix; 

apply  Theorems  3.1  and  3.2  which  give  conditions  for  a  Markov  chain  to  possess  a 

unique  stationary  distribution  and  to  have  a  limiting  distribution; 

classify  the  states  of  a  finite  Markov  chain; 

classify  the  states  of  an  infinite  Markov  chain  in  simple  cases; 

use  generating  functions  to  obtain  a  relationship  between  the  return  probabilities 
P'u  and  the  first  return  probabilities  f\H)  for  a  state  /  of  a  Markov  chain; 

find  stationary  distributions  of  Markov  chains,  and  use  the  Basic  Limit  Theorem 
for  a  recurrent  irreducible  aperiodic  Markov  chain  to  find  the  mean  return  times 
of  the  states. 


Appendix:  Solutions  to  questions 


Section  1 

1.1  The  result  is  obtained  by  applying  the  rule 
P(A  n  B)  =  P(A  |  B)  P{B)  in  a  similar  way  to  the  derivation  of 
Result  (1.4)  and  assuming  throughout  that  the  process  starts 
in  state  z  and  so  the  condition  [AT0  =  z]  holds. 

Let  A  =  [ X„  -----  z]  and  B  =  [Zt  =  Xn_2  =  y];  then 
P{X,  =  j,X2  =  k,  ...,  =  y,X„  =  zj X0  =  z) 

=  P(Xn  =  z\Xl  =  j,...,Xn_1=y,X0  =  i) 
x  P(X,  =j,...,Xn.l=y\X0  =  i) 

—  P(xn  —  z\xn-i  =  y)P(Xi  — j,  ■■■,  xn_i  =  y|  A’o  =  z), 
applying  Condition  (1). 

By  a  similar  argument, 

P{Xx  =j,...,Xn-2  =  x,Xn.l=y\X0  =  i) 

—  P(X„~i  =  y  \Xn-2  =  x) P{X1  =  j, X„-2  =  x  |  -^o  =  0- 
Applying  the  argument  repeatedly  leads  to 
P(Xl=j,...,Xn  =  z\X0  =  i) 

=  P(Xn  =  z\Xn.l  =y)P(Xn_l  =y\Xn„2  =  x)... 

...  x  P(Xi  =  ;| X0  =  z) 

=  PyzPxy  ■  •  ■  Pij 
=  PijPjk  •••  Pyz- 

1.2(i)  Gambling  against  a  casino  was  discussed  in  the 
audio-tape  session  in  Unit  5.  The  state  space  is  infinite  in  this 
example;  the  transition  matrix  is 

0  12  3  4 

0  [  10  0  0  0 

1  q  0  p  0  0 

2  0  q  0  p  0 


(ii)  This  is  the  random  walk  with  two  reflecting  barriers 
that  was  used  for  the  gambling  scenario  in  Subsection  5.1  of 
Unit  5.  The  transition  matrix  is 

1  2  3  4  5  .  a  - l 

1  q  p  0  0  0 

2  q  0  p  0  0 

3  0  q  0  p  0 


a  —  2  0 

a- 1  0 


1.3  The  diagram  is  shown  in  Figure  S.l  below;  it 
represents  a  simple  random  walk  with  one  absorbing  barrier 
and  one  reflecting  barrier. 

So  the  state  space  is  {1,2,3, ...,  20}  and  the  transition  matrix 
is 

1  2  3  4  5  .  20 

1  Yq  p  0  0  0  01 

2  q  0  p  0  0  0  1 

3  0  q  0  p  0  0 


19  0  0  q  0  p 

20  [o  .  0  0  0  1 

1.4(i)  If  z  of  the  six  containers  are  occupied,  then  the 
probability  that  the  next  ball  is  put  into  an  occupied 
container  is  i/6,  so  p„  =  i/6;  the  probability  that  it  is  put  into 
an  unoccupied  container  is  1  —  i/6,  so  pi  i+1  =  1  -  i/6.  These 
probabilities  are  shown  in  Figure  S.2  below. 

The  transition  matrix  is 

0  1  2  3  4  5  6 

0  [0  1  0  0  0  0  0" 

1  0  £  |  0  0  0  0 

2  00ii000 

3  000ii00- 

4  0  0  0  0  j  )  0 

500000ii 

6  |_0  o  0  0  0  0  1_ 


Figure  S.l 
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(ii)  For  this  example,  it  is  convenient  to  arrange  the  states 
in  the  diagram  in  the  same  configuration  as  the 
compartments  in  the  maze. 


The  transition  matrix  is 

1  2  3  4  5  6  7  8  9 

1  Toioooiooo" 

2  iOiOiOOOO 

3  O^OiOOOOO 

4  00*0^000$ 

5  0  i  0  i  0  ±  0  i  0 

6  iOOOiOiOO 

7  0  0  0  0  0  i  0  i  0 

8  0000^0*0$ 

9  l  OOOiOOO^O 


1.5  We  have 

_(m  +  n) 

rij 

=  p(Xm  +  n  =j\X0  =  i),  by  definition, 

=  Z  =  k  |  *0  =  /)  P(Xm+m  =  j  I  =  k  and  X0  =  i), 

keS 

by  the  Theorem  of  Total  Probability, 

=  Z  pWkj, 

keS 

using  Condition  (2)  and  the  Markov  property  (1A)  to 
obtain  ptfj. 

1.6(i)  With  0  for  dry.  1  for  wet.  we  have 
0  1 

0  (~ 0.750  0.250] 

1  L.0.338  0.662J" 

(ii)  The  required  probability  is  p\\0),  so  we  must  find  P10. 
We  have 


P2 


0.647  0.353“ 
0.477  0.523_  ’ 


P3  =  PP2  = 


p5  _  p2p3  . 
pi  0  __  p5p5 


0.580  0.420] 
0.568  0.432]' 
“0.575  0.425] 
0.575  0.425 J 


0.605  0.395’ 
0.534  0.466]  ’ 


(Alternatively,  you  may  have  calculated  P2,  P4,  P8  and 
hence  P10.) 


Thus  the  probability  that  if  it  rains  on  1  January,  it  will  rain 
on  11  January,  is  0.425.  Notice  that,  to  three  decimal  places, 
this  is  also  the  probability  that  it  will  rain  on  1 1  January  if  it 
is  dry  on  1  January. 


1.7(i)  In  this  example,  the  Markov  property,  Condition  (1), 
says  that  a  man’s  occupational  status  depends  only  on  his 
father’s  status  and  not  on  his  grandfather’s  status.  In 
Condition  (2),  we  are  implicitly  assuming  that  the  factors 
influencing  transitions  in  occupational  level  between 
generations  remain  the  same  from  generation  to  generation. 
(This  may  not  be  a  reasonable  assumption  in  a  rapidly 
changing  society.) 

(ii)  The  required  probability  is  P32i  =  (P2)3i>  which  is 
0.01  x  0.45  +  0.50  x  0.05  +  0.49  x  0.01  =  0.0344. 

Section  2 

2.1  Applying  the  Theorem  of  Total  Probability  to 
a[ f + 1(  =  P(Xn+1  =j),  conditioning  on  X„,  we  have 

P(Xn  +  i  =j)=£P(Xn+l  =j\X„  =  i)P(Xn  =  i). 

ieS 

So,  for  each  jeS, 

<4+"  =E4V 

ieS 

Combining  these  expressions  for  ^n+1)  into  a  row  vector,  we 
have 

a(n  +  1)  =  a<n)P. 

2.2  We  shall  use  Equation  (2.2)  three  times,  for  n  =  2, 
n  =  3  and  n  =  4;  this  gives 

W  tf’]  -[»*[]  t 

L*  4_. 

=  [t&  its]  =  [0.602  0.398], 

correct  to  three  decimal  places.  Then,  to  the  same  accuracy, 

K4)  a<4)]  =  [0.602  0.398]  0  5  °'5  =  [0.600  0.400], 

LO.75  0.25] 

la(o5)  «(,5)]  =  [0.600  0.400]  =  [0.600  0.400]. 

The  probabilities  that  the  student  buys  brand  A  after  3,  4 
and  5  days  are  0.602,  0.600  and  0.600  respectively;  the 
corresponding  probabilities  for  brand  B  are  0.398,  0.400  and 
0.400. 

2.3  The  initial  probabilities  can  be  interpreted  as  the 
proportions  of  male  wage-earners  in  a  particular  generation 
in  each  of  the  three  occupational  groups.  The  absolute 
probabilities  are  the  predicted  proportions  in  the  three 
groups  in  succeeding  generations. 

2.4  The  first  interpretation  of  nt,  the  limiting  probability 
for  state  1,  is  as  the  long-run  proportion  of  male  wage- 
earners  in  a  generation  who  are  in  the  upper  occupational 
level.  The  second  interpretation  is  as  the  long-run  expected 
proportion  of  generations  of  any  given  family  in  which  the 
male  wage-earner  is  in  the  upper  occupational  level.  (This 
interpretation  is  valid  if  it  is  assumed  that  the  model  is 
unchanged  for  a  large  number  of  generations,  and  this  is  not 
likely  to  be  the  case;  for  example,  considerable  changes  in 
job  mobility  patterns  have  occurred  in  the  last  100  years.) 

The  limiting  probabilities  n2  and  n3  are  interpreted  in  a 
similar  way  for  the  middle  and  lower  occupational  levels. 

2.5  p<n+l)_p(n!p 

pjOfi  i] 

j «  Mfj  i  ij 

So.  multiplying  these  two  matrices  together,  we  obtain 
Poo  11  =  Poo  X  I  +  Pol  X  I 

-  M  x  i  +  (l  -  P$)  X  | 

=  I  —  ip oo,  as  required. 


41 


2.6  The  limiting  distribution  n  =  [7t0  7t,]  must  satisfy 
*o  +  *1  =  1  and  the  equations 

*o  =  0  -  *)*o  +  /?*!, 

*,  =  y.n0  +  (1  —  /i)7t , . 

Discarding  the  last  equation,  we  must  solve  the  simultaneous 
equations 

*o  =  (1  ~  +  /?7t  i  and  710  +  71!  =  1. 

So  n0  satisfies 

*o  =  (1  “  «)*o  +  £0  ~  *o)- 
That  is,  (a  +  P)n0  =  P,  so  that 
*o  =  /J/(a  +  /?); 
and,  since  7i0  +  7t,  =  1, 

*i  =  a/(x  +  /?). 

2.7(i)  For  Example  1.4,  a  =  0.174  and  /?  =  0.600,  so 
*o  =  /VO*  +  P)  =  0.775  and  tt,  =  1  —  7r0  =  0.225,  correct  to 
three  decimal  places. 

(ii)  For  Example  1.5,  a  =  0.250  and  p  =  0.338,  so 
*o  =  +  P)  =  0.575  and  7t!  =  1  —  7r0  =  0.425,  correct  to 

three  decimal  places. 

2.8  The  limiting  distribution  n  =  [71,  7i2  7t3]  must  satisfy 
*1  +  7i 2  +  7t3  =  1  and  any  two  of  the  three  equations 
7r,  =  0.571,  +  0.27r2  +  0.1tc3, 

7T2  =  0.471,  +  0.67T2  +  0.2tI3, 

7t3  =  0.1 7c,  +  0.2n2  +  0.77t3. 

Discarding  the  last  equation  and  substituting 
*3  =  1  —  7t,  —  7i2  in  the  other  two  equations  gives 
0.671,  —  0. 1  tt2  =  0.1, 

—  0.27c,  +  0.67t2  =  0.2. 

These  have  the  solution  71,  =  4/17  ~  0.235, 

*2  =  7/17  a  0.412,  and  hence  tt3  =  6/17  ~  0.353.  (The  values 
of  71 2  and  n3  are  not  required.)  The  proportion  of  male  wage- 
earners  in  the  upper  occupational  level  in  the  long  run  will 
be  0.235. 

Section  3 

3.1  (i)  A  diagram  for  this  Markov  chain  is  given  below. 


The  communicating  classes  are  {0, 1}  and  {2};  they  are  both 
closed. 

(ii)  This  Markov  chain  has  the  following  diagram. 


0  1  - » -  2  3 


The  communicating  classes  are  (0, 1},  which  is  not  closed, 
and  {2,3},  which  is  closed. 

3.2  A  diagram  for  the  gambler’s  ruin  is  given  below. 


3.3(i)  Question  3.1  (i):  This  Markov  chain  has  two  distinct 
closed  classes,  so  it  does  not  have  a  unique  stationary 
distribution. 

Question  3.1(ii) :  This  Markov  chain  has  only  one  closed 
class,  so  it  does  have  a  unique  stationary  distribution. 
Question  3.2:  The  gambler’s  ruin  Markov  chain  has  two 
closed  classes,  {0}  and  {a},  so  it  does  not  have  a  unique 
stationary  distribution. 

(ii)  Question  3.1  (i):  n  =  [7t0  tc,  tt2]  must  satisfy  7t  =  7rP 
and  7r0  +  7t,  +  7r2  =  1.  That  is, 

*0  =  i*0  +  I7T,  , 

*1  =  t*o  +  i*l» 

7t2  =  71 2 

and 

*0  +  7t,  +  7I2  =  1. 

So  7r0  =  7T,  and  7t0  +  7t,  +  7i2  =  1.  Possible  solutions  are 
[i  2  0]  ar>d  [0  0  1],  for  example.  Any  solution  of  the  form 
7i  =  [x  x  1  —  2x],  with  0  <  x  <  is  a  stationary 
distribution  for  the  Markov  chain. 

Question  3.1  (ii):  n  =  [7i0  tt,  tt2  tc3]  must  satisfy  n  -  nP 
and  7t0  +  tt,  +  7r2  +  ?r3  =  1.  That  is, 

7to  =  4*0  +  ^Tt,  , 

*1  =  47to  +  T7t,  , 

*2  =  +  4*2  +  i*3> 

*3  =  1*2  +  ^*3 

and 

*0  +  *1  +  *2  +  7I3  =  1. 

The  first  two  equations  give  7t0  =  tt,  =  0.  Then  the  third  and 
fourth  equations  simplify  to  n3  =  3*2,  and  from  the  last 
equation  we  obtain  n2  =  0.4,  n3  =  0.6.  So  this  Markov  chain 
has  a  unique  stationary  distribution  n  =  [0  0  0.4  0.6]. 

Question  3.2  (The  gambler's  ruin):  n  =  [7r0  7r ,  ...  na]  must 
satisfy  7r  =  ttP  and  7i0  +  7t,  +  . . .  +  7ia  =  1.  That  is, 

*o  =  *o  +  <?*i,  which  gives  n,  =  0, 

*i  =  <?*2>  which  gives  n2  =  0, 

*2  =  p*,  +  qn3,  which  gives  7r3  =  0. 

Similarly,  *4,  ...,  *fl_,  are  all  zero. 

Then  from  *0  +  *,+...  +  7ra  =  1, 

*o  +  *a  =  1  • 

Possible  solutions  are  7i  =  [1  0  0  ...  0  0]  and 

*  =  [0  0  ...  0  0  1];  any  solution  of  the  form 

jt=[x  0  0  ...  0  0  1  —  x],  where  0  <  x  <  1,  is  a  stationary 

distribution  for  the  Markov  chain. 


3.4  A  diagram  for  the  Markov  chain  is  given  below. 


Returns  to  state  0  can  occur  after  any  number  of  transitions, 
so  state  0  is  aperiodic.  Returns  to  state  1  and  returns  to  state 
2  can  occur  only  after  an  even  number  of  transitions;  states  1 
and  2  are  periodic  with  period  2. 


The  communicating  classes  are  {0},  (1,2,  ...,  a  —  1}  and  {a}; 
{0}  and  {a}  are  closed,  (1,2, ...,  a  -  1}  is  not  closed. 
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Section  4 

4.1  Figure  1.2,  which  illustrates  this  Markov  chain,  is 
repeated  below. 


i  2 

"  T  I 


State  3  is  an  absorbing  state,  so  it  is  recurrent.  State  0  is 
transient,  since  the  chain  is  certain  to  leave  state  0  and  never 
return.  For  state  1,  the  probability  of  leaving  and  never 
returning  is  f ,  and  for  state  2  this  probability  is  i,  so  states  1 
and  2  are  also  transient. 

4.2  Question  3.1(i ):  {0,1}  and  {2}  are  both  closed  classes, 
so  all  the  states  are  recurrent. 

Question  3.1  (ii):  {0, 1}  is  not  closed  and  {2,3}  is  closed,  so 
states  0  and  1  are  transient  and  states  2  and  3  are  recurrent. 


Section  5 


5.1(i)  The  first  return  to  state  0  occurs  after  1  transition  if 
(starting  from  state  0)  the  first  trial  results  in  a  failure,  so 

/oo)  =  <?• 

For  n  >  2,  the  first  return  to  state  0  occurs  after  n  trials  if 
the  first  n  -  1  trials  are  successes  and  the  nth  is  a  failure,  so 

fM  =  pn-'q. 

(ii)  Now  /00,  the  probability  of  ever  returning  to  state  0,  is 
given  by 

/oo  =  /oo  +  tiV  +  foo  +  ... , 
so 

/oo  =  q  +  pq  +  P2q  +  ... 

=  q(l  +  p  +  p2  +  ...) 


So  state  0  is  recurrent. 


5.2(i)  The  return  probability  p\2)  is  the  probability  that, 
starting  from  state  i,  the  Markov  chain  is  again  in  state  i 
after  two  transitions.  This  event  can  happen  in  two  ways: 
either  the  chain  returns  for  the  first  time  after  one  transition 
and  then  returns  again  after  one  more  transition  or  the  chain 
returns  for  the  first  time  after  two  transitions.  These  two 
events  are  mutually  exclusive,  so  p\f}  is  equal  to  the  sum  of 
their  probabilities.  The  first  event  has  probability  /{'Vi/  and 
the  second  has  probability  f\2\  so 

PW  =fu)P{,i)  +  f&2). 

Since  p\f]  =  1  and  /}0)  =  0,  we  can  rewrite  this  as 

Similarly,  there  are  three  ways  in  which  a  return  to  state  i 
can  occur  after  three  transitions;  the  chain  can  return  for  the 
first  time  after  one,  two  or  three  transitions.  If  the  first  return 
occurs  after  one  transition,  then  another  return  must  occur 
two  transitions  later;  and  if  the  first  return  occurs  after  two 
transitions  then  another  return  must  occur  after  one  more 
transition.  Thus  the  three  events  have  probabilities 
/i2)Pi/)  and  /•‘•3)  =  fVygK  and 

P'P  =fu)Pif)  +/!-2)pL-)  +/i3)Pi,?> 

=/ii0)Pi.?)  +/11)p}2)  +fii2)p\P  +/<-3,pL0)- 


(ii)  There  are  n  ways  in  which  a  return  to  state  i  can  occur 
after  n  transitions;  the  chain  can  return  for  the  first  time  after 
1,2,3, ...  or  n  transitions.  By  an  argument  similar  to  that  in 
part  (i),  these  n  events  have  probabilities  /!1)p}"_l),  f\2)p\"~2\ 
•••> /in)Pi?);  so,  since /j0)  =  0,  we  have  (for  n  >  1) 

P!/  =ffW  +/,r)pi-r  ■ 11  +  ...  +  /<">p}°> 


5.3(i)  For  n  >  1,  the  term  in  s"  in  the  product  Fu(s)  Pu(s)  is 
/•!0)  x  +/}1)s  x  P\r  V1  +  ...  +  f\n)sn  x  p|?> 

=  (/•i0)Pi">  +/!1)p‘r1)  + ...  +/fpifv 
=  p\l}sa,  using  Solution  5.2(h); 
this  is  the  term  in  s"  in  Pu(s).  The  constant  term  on  the  left- 
hand  side  of  Equation  (5.1)  is  p}f)  =  1  and  the  constant  term 
on  the  right-hand  side  is  1  +/}0)p|°),  which  is  also  equal  to  1 
Hence,  the  two  sides  of  Equation  (5.1)  are  equal,  thus 
establishing  the  identity. 


(ii)  Rearranging  Equation  (5.1),  we  have 

F“(s>='-m 

oo 

or  equivalently,  writing  X  for  £  , 

Letting  s  increase  to  1  in  this  last  expression,  we  obtain 
LJu  ZPm’ 

So  fa  =  X f\H)  =  1  if  and  only  if  Xp!"’  is  infinite. 

(This  derivation  of  the  criterion  for  recurrence  of  state  i  does 
not  constitute  a  rigorous  proof.) 


5.4  The  mean  return  time  to  state  1  could  be  found  by  the 
method  used  to  find  the  mean  return  time  to  state  0;  but  it  is 
much  quicker  to  use  symmetry  to  obtain  p,  from  p0. 
Replacing  a  by  /J  and  /?  by  a  in  the  expression  for  p0  gives 
Pi  =  (/  +  a)/a;  so  the  mean  time  between  wet  days  is 
(0.600  +  0. 1 74)/0. 1 74  or  approximately  4.45  days. 

5.5(i)  For  Example  2.1,  n0  =  0.6  and  tc{  =  0.4,  so 

Po  =  1/tio  -  1-67  days.  On  average,  the  student  buys  brand  A 

every  1.67  days. 

(ii)  For  Example  2.2,  n0  =  0.432,  so  7r,  =  1  -  n0  =  0.568 
and  p,  =  l/fl,  ~  1.76.  The  average  number  of  letters  from 
one  consonant  to  the  next  is  1.76,  approximately. 

(iii)  For  Examples  1.6  and  2.3,  7r,  ~  0.062,  so 

Pi  =  l/^i  —  16.1.  The  mean  number  of  generations  between 
male  wage-earners  in  the  upper  occupational  level  (in  a 
family)  is  16.  (However,  the  model  is  not  likely  to  be  valid 
over  such  a  long  period  (see  Solution  2.4),  so  in  practice  care 
must  be  taken  in  applying  a  result  such  as  this.) 

5.6(i)  We  require  a  solution  of  n  =  nP  and 
7t0  +  Jr,  +  7i2  =  1;  that  is,  of 

7t0  =  0.37to  +  0.l7tj  +  0.l7t2,  6-2- 
Jti  =  0.1tto  +  0.3^!  +  0.  l7t2, 
n2  =  0.67to  +  0.671!  +  0.87t2, 

satisfying  n0  +  Hi  +  n2  =  1.  Discarding  the  third  equation 
and  substituting  7t2  =  1  —  jiq  —  in  the  first  two  equations 
gives 

7r0  =  0.27ro  +  0.1  and  =  0.27c,  +  0.1, 
so  that  7t0  =  £  and  nl  =  £.  Since  7t2  =  1  —  7t0  —  nx ,  we  have 
rc2  =  j.  The  proportion  of  trees  along  the  path  that  are 
healthy  is  approximately  +  n2  = 

(ii)  By  Theorem  5.1,  p0  =  l/7i0  =  8;  so  on  average  every 
eighth  tree  is  diseased  and  there  are  (on  average)  seven 
healthy  trees  between  one  diseased  tree  and  the  next. 
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5.7(i)  The  transition  matrix  for  the  Markov  chain  is 


P  = 


q  p  0  0  0  ... 

q  0  p  0  0  ... 

q  0  0  p  0  ... 


The  stationary  distribution  n  =  [rc0  n2  ...]  satisfies 
n  =  7rP;  that  is,  [>0  nv  n2  ...]  =  [ n0  n1  n2  ...]P;  so,  for 
•k>  1, 


nk  ~  nk-lP- 
Hence,  for  k  >  1, 
nk  =  n0pk. 

Also, 

+  n{q  +  n2q  +  ... 

=  q(n0  +  TTj  +  n2  +  ...)  =  q, 
so,  for  k  >  0,  nk  =  qpk. 

(ii)  The  mean  number  of  trials  between  success  runs  of 
length  at  least  k  is 

Pk  =  1/"*  =  1  /(qpk),  for  k  >  0. 
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